I will likely go back to most of the parts of this post in more details later but, let's have a quick overview of what you need to know to start.
Keep it simple and have fun
The stack is the set of tool / language / technologies that are the building blocks of your tools.
I don't have a strong opinion of what to use. Everybody has its own preference and it may also depend on your need.
For a web based application, here is my current suggestion: use Laravel + PHP + TailwindUI and related ecosystem. Even if PHP may not be as hip as other languages, it's proven, reliable and a lot of support.
You'll likely need a database for data storage. Simple files, SQLite or MySQL are good options to get started depending on what you build and need.
Choose Git
Hosted on Github, Gitlab, or Bitbucket.
How you structure it depends on what you want, see what your developer suggest.
Keep it simple.
I am focused more on web applications, and that requires you to have things running on a server.
Where it lives depends on what you need. You have endless options ; one your computer, your own server, shared hosting, VPS, on the cloud, Azure, AWS, DigitalOcean, Serverless, ... The options are endless.
The management of it can be annoying.
If you are using my stack above, use laravel forge to handle deploying code and sinply be done with it until you need more.
You may want to be sure things are running correctly. If you are only having things through a UI frontend you access, you will know when things are not working. In most cases, you will have some background processes, things that runs independently, without you knowing, to perform multiple tasks for you.
Anything that can or may take time to finish. In that case, you need a solution to monitor that everything runs successfully. Set up logs to be able to investigate and fix issues quickly.
You may also need to monitor your infrastructure / servers themselves ensuring that they have enough resources to do. That is slightly less important to have right from the beginning unless it is facing external users or people you don't interact with and rely on your tools. The main goal is to prevent outages.
You can look into Zabbix, Datadog, Sentry, Oh Dear, Telescope, Horizon, ... whatever that can observe what's going on and alert you when something fails (or is about to).
That is something important, make sure that you won't loose anything, and more importantly, that you can recover from any outage or issue quickly.
The way to go will depend on your stack and where you run the different parts of your application.
Version Control covers your code. Then, database and user submitted assets may have to be covered.
Important note, your backups are only as good as your ability to recover. Make sure you have a recovery plan as soon as possible, before any issue happens.
Don't store backups on the same location as your main application (the further it is the better).
Ideally, everything you do would be relying on the latest version of everything. It won't. Do your best to keep up, but don't overdo it either. I like to be updated on everything, but, the more dependencies you had, the less you will be able to be on top of all updates.
Not only because you may have a lot to handle, but also, because there may be breaking changes, incompatibility and conflicts preventing you to.
If your tools are not external facing, you have less needs to be on top of things. But, don't get too much behind either.
The most important : stay on top of security updates and make sure you're on supported versions for the main components of your stack.
There will be headaches, worries and unexpected challenges.
Focus on making progress, 80/20 way, get results quickly and keep the wheel rolling
Make it a fun process.