The Charles Town Race Track opened on December 2, 1933, shortly after West Virginia legalized racing and parimutuel betting. The first track in the state, the Jefferson County complex contained 22 buildings and included 12 stables, a clubhouse, 44 betting windows, a 3,000-seat, steam-heated grandstand, and a 200-seat restaurant.

The 1933 and 1934 racing seasons were financial failures, and the new track went into receivership. Then Albert J. Boyle took over, putting up new money and convincing the creditors that he could make a go of it by pulling bettors from Washington and Baltimore and by promoting racing during the winter season. Boyle succeeded, and for more than 20 years he was a dominant figure in West Virginia racing.

By 1959, Shenandoah Downs, a competing race track, had opened nearby. One of the first tracks on the East Coast to be built for night racing, Shenandoah Downs competed for racing days, and profits, with the older Charles Town Race Track, whose facility could support only daylight meets. The owners of Maryland's Pimlico Race Course bought the Charles Town Race Track in 1959 and began extensive renovations. By 1964, they had installed lighting that enabled them to compete with the newer track.

Shenandoah Downs closed in 1976, with Charles Town picking up its racing days and doubling receipts. The Kenton Corporation bought both tracks in 1978. Sunday racing was approved in a 1979 referendum. In 1983, the Charles Town Race Track was sold to the Rapid America Corporation, a group of 15 businessmen that included 11 local investors.

With the future of the operation in doubt, Penn National Gaming agreed to purchase Charles Town Races if Jefferson County approved the installation of video lottery machines. The vote passed on November 5, 1996, and two months later, Penn National bought the race track, refurbishing the thoroughbred racing facilities. On September 10, 1997, the new owners opened a gambling center with 220 operating video lottery machines. A multi-million dollar simulcast center was added in January 1998. Like other West Virginia race facilities, Charles Town now relies extensively on non-racing income. The video lottery portion of the operation had been expanded to over 5,000 video slots by 2010. On December 5, 2009, Jefferson County voters approved an expansion of gambling to allow table games at the race track. Charles Town Racing and Slots changed its name to Hollywood Casino at Charles Town Races and began offering table games to patrons in summer 2010. Sports betting was added in 2019.