Let me tell you how I ended up at this point where I started using Secret Manager,
I was building my portfolio application (summy.dev) and I wanted to use some 3rd party apis for my application and I wanted to store the access tokens and as a common joe I was storing it in environment variables while the application was in development phase, and one fine day I wanted to take it live and I had my infra set up at google app engine and my ci/cd via github action.
I broke my back trying to figure out how to inject environment variables to my google app engine in a safe and secure manner, a lot of digging went through and all I could discover wasn't much of safer options, some suggested to having it over the yml config of app engine and to inject into environment from there or to have it in my CI/CD process to inject the environment variables, all these methods had one thing in common - "Have my secret access token in code repository" , nope , that's not safe right ? or how about setting up environment variables directly in app engine, sounds safe enough to me but app engine does not support that, whaaat ?! , yes, I too had the same expression, "How can it not support something as crucially required as environment variables for secrets storage !" , then I went through some videos over youtube from google to understand the "WHY" part of it [watch it here] . I'd highly recommend you to watch this video.
The main key difference between using Secret Manager v/s Environment Variables can be split up into three key factors - Level of Security, Flexibility, and Management Capabilities
Security
Environment Variables -
Well I do agree that environment variables are pretty convenient to use especially for storing sensitive information such as API Secret tokens, DB access keys etc but the main draw back of it is being env vars (environment variables) can potentially be accessed by anyone with access to the system or through logging mechanisms.
Secret Manager -
The best thing about this is they provide a secure means to store secrets by encrypting them at rest and in transit, along with that it provides other value added services like fine grained access control, auditing, rotation policy etc., enhancing overall security.
Flexibility
Environment Variables -
They are typically straight forward to use and it can be used in applications without any additional setups to be done but the catch is, it becomes nerve wrecking when you have to maintain large number of keys multiplied by different/multiple environments.
Secret Manager -
Best thing about this is it offers more flexibility when it comes to managing secrets centrally, it allows to store not just key-value pairs but as well complex data structures and it tops it up by providing APIs and SDKs for accessing secrets programatically, making integrations with your apps much easier.
Management
Environment Variables -
Env vars might be quickest way to get started but easily it becomes tough as it often involves manual process such as updating configurations on each system individually which might result in inconsistencies and potential security risks if not handled properly since it involves human intervention.
Secret Manager -
Since it is centralized management for secrets allowing us to easily CRUD secrets through its interface or APIs and also it supports neat features like automatic rotation of secrets, helping us maintain secrets without human intervention.
summary, for storing secrets - secret manager is the best answer.
Secret manager services are available with Google Cloud Program as well as with AWS.
p.s - feel free to reach out to me if you need hand in setting up secrets manager in your application !