For investors worried about how the election will impact their portfolios over the long haul, fear not: Elections have seldom had a lasting impact on equity prices.

President Donald Trump has warned that the stock market will crash if former Vice President Joe Biden wins the presidential election. Some market experts have also raised concern about the potential for a "blue wave" if Democrats gain a majority in the Senate, win the White House and keep control of the House.

However, history shows that stocks usually do well regardless of which party controls the White House or Congress. 

"I think people overestimate the importance of politics for investing," said David Kelly, chief global strategist at J.P. Morgan Asset Management. 

Are Republicans or Democrats better for stocks? 

Data over the past 78 years shows that party control over either chamber has relatively little to do with long-term changes in the broad S&P 500 stock index.

Starting in 1942, the numbers indicate that Republican and Democratic majorities in the House and Senate have had little impact on stock prices in the two years following an election. 

The same holds true when you look at the number of party seats gained or lost in the House and Senate, against stock prices in the S&P 500 during that period. 

The data yields similar results for the November to November cycle, which is a gauge of market sentiment to the election, as well as January to January, which shows the actual market performance of the Congress. 

Presidents and stocks 

Where you start to see more of an impact is the combination of party control in both chambers of Congress. 

Data compiled by LPL Financial shows that beginning in 1950, the average annual stock return was 17.2% under a split Congress, 13.4% when Republicans held both chambers, and 10.7% when Democrats had control.

LPL Financial's Ryan Detrick said in a note that "markets tend to like checks and balances to make sure one party doesn't have too much sway," hence the stronger stock performance during a split Congress.

But when you broaden it out even further to consider the party of the president in tandem with party control of the two chambers, the trend of a split Congress being best for stocks doesn't always hold true. 

Sam Stovall, CFRA chief investment strategist, looked at how the market has performed under six political scenarios: a White House and Congress all under the same party, a White House with a split Congress, and a White House and Congress hailing from two different parties. Stovall included election data going back to 1945.

Of all the possible combinations, stocks appear to perform best when a Democrat is in the White House and the Congress is split. The second highest returns happen when a Democrat is president and Republicans control the Congress.

But ultimately, Stovall said, investors should be wary of reading too much into these numbers. 

"It's a good example of how you can have data tell whatever story you want," he said. "If you want to favor the Democrats, talk about the presidency. If you want to favor the Republicans, talk about House control."

Bob French, director of investment analysis at McLean Asset Management, agrees. "We can go in and slice and dice the data however we want and most of the time come up with whatever answer we want."

However the vote plays out Tuesday, Fundstrat's Tom Lee thinks the stock market is poised to take off.

"At least 90% of [our] portfolio strategy would be identical under either win," Lee said in a note on Oct. 6. In either case, Lee predicts the outcome of the election will be bullish for stocks.