You are very fortunate. Metallurgy and plastics have come a long way since the days of JMB and the classic designs. You have the advantage of having space-age lightweight materials to work with which have amazing tensile and compressive strength attributes unheard of only about 50 years ago.
However, that being said, you still need to have at least a working knowledge of engineering to be able to understand why certain aspects of your design either a) won't work, or b) have already been tried and won't work.
Finally, you need a much better knowledge of the law than you demonstrate.
For instance -- you're talking about an open bolt design. Were you aware that since about 1988, open bolt semi-auto designs are prohibited for civilian use by the ATF? Because they determined it was too easy to "convert" such designs to fully automatic? So, open bolt is out.
As far as a "new" recoil system: Are you familiar with what has already been done and how? Take a look at the Browning Auto 5 for a shotgun recoil system designed by a master with true engineering skills and NO advantages with modern metals (The Auto-5 was designed by JMB back in 1898, patented 1900, and in production until 1998 - 100 years.)
I know you're eager and excited, but your question is full of ego and not much technical knowledge. I'd say your first step is to, as I said before, STUDY WHAT IS ALREADY IN EXISTENCE. If you're designing a shotgun, then you study the Auto-5, the Remington 1100, the Remington 11-48, the Saiga-12, the Akdal MKA-1919, and even the Atchisson Assault 12 if and when you can find pictures of it, and its modern redesign, the AA-12. Understand innovations and derivative works -- like the difference between the AA-12 and the USAS-12 -- and what works and what doesn't. Especially of intrest could be the Intrepid Tactical AR-12.
Along the way, you definitely need to study Federal law, specifically Title 18 of the United States Code as it pertains to firearms, and Title 26 of the United States Code as it pertains to firearms.
With all of this data, then you study your ammunition, in which case I'm talking about the 12-gauge round. Dimensions. Powder charge. Velocity. Cartridge composition -- what kind of brass? What size is the primer pocket? What material is the hull made of usually? What kind of shotgun wads are available? And most importantly: What dynamic range of loadings, from birdshot to slugs, are available in the 12-gauge configuration and their pressures and velocities, are available?
Then, decide what you want your gun to shoot...all of them? Just the combat loads? Specific ammo?
Understand that your decisions along the way will affect your marketing of this product, and your target sales demographic.
Once you understand the AMMO and the pressures involved, start considering the physics of firearms materials. How thick does your chamber have to be to safely contain those pressures? What margin of safety do you want to build into your shotgun? What materials will you use to contain those pressures? How will you design the tooling to make your receivers/chambers/barrels/other parts efficiently?
Which means you have to go back to knowing what already exists. JMB didn't have CNC machines. His designs were at the start of the industrial revolution, and among other things, he was a machinist. You have the advantage of expert machinists with industrial machinery, presses, mills, CNC 3-axis machines, waterjets, and, as I have just recently mentioned, laser sintering in stainless steel.
Just coming up with a design that looks cool is not "Designing." Don't get me wrong, it's a good start. But don't go patting yourself on the back just yet. You just did the fun part. Now it's time to do your homework so you have something to at the very least, EXPLAIN to an engineer, both so you can actually patent something PATENTABLE and so you can have an engineer take your design from the drawing board and turn it into a working prototype.
Good luck. If your design is all you say it is, I look forward to seeing it in the marketplace.