[Feature Request]: Add Support for Sharing lifecycleEnvironments Directly in Project Configuration Files

What would you like to happen?

Currently, Apache Hop lacks the capability to share lifecycleEnvironments along with the project
configuration files in a way that ensures consistency across all developers.

When a new environment is added or modified for a project, the changes are made in the hop-config.json file.
This file is user-specific and not shared through the project repository, leading to the following issue:

Manual Updates: Developers need to manually update their local hop-config.json files to reflect changes made by others, which can be cumbersome and error-prone.

I'd like to request the addition of a feature that allows lifecycleEnvironments
to be linked directly to the project configuration files (project-config.json or a similar project-specific file).
This would involve:

Integration: Storing lifecycleEnvironments in a project-specific configuration file that is version-controlled and included in the project repository.
Automatic Synchronization: Ensuring that changes to environments are automatically reflected for all developers working on the project, without requiring manual updates to individual hop-config.json files.

See also: https://chat.project-hop.org/hop/pl/f936cqzr5b8fj8fzccxpzod3no

Issue Priority

Priority: 2

Issue Component

Component: Hop Config

Any plans on this feature? We would really like to be able to store environments and their json files not in hop-config.json, but project-config.json!

work in progress if I'm not mistaken.

Great to hear that! In the course of some other ticket?

.take-issue

@mattcasters

A lifecycleEnvironment is:
name, purpose, projectName, canvasText, configurationFiles, attributesMap

projectName means it belongs to exactly one project, and configurationFiles are
${PROJECT_HOME}/... paths in practice. So the record holds nothing user- or
machine-specific — unlike projectConfigurations.projectHome, which is an absolute local
path. Environments are the only project-bound block in projectsConfig that is already
portable, and they sit in the one file that can never be shared: hop-config.json also
holds guiProperties (DarkMode, fonts, icon size), azure, googleCloud, dropbox,
gitConfig and 60+ variables.

@mattcasters — credentials don't live in the environment, they live in the config files
it points to. Sharing an environment shares which files to load, not what is in them.

Proposal: two layers, not a move

The environment list currently exists exactly once. Moving it to the project would make it
fully shared and remove per-developer settings. Instead, merge two sources when a project
is opened.

project-config.json (committed, same for everyone):

{
  "metadataBaseFolder" : "${PROJECT_HOME}/metadata",
  "unitTestsBasePath" : "${PROJECT_HOME}",
  "dataSetsCsvFolder" : "${PROJECT_HOME}/datasets",
  "enforcingExecutionInHome" : true,
  "autoExportMetadataFilename" : "metadata.json",
  "parentProjectName" : "default",
  "config" : {
    "variables" : [ ]
  },
  "lifecycleEnvironments" : [ {
    "name" : "development",
    "purpose" : "Development",
    "configurationFiles" : [ "${PROJECT_HOME}/config/dev.json" ]
  }, {
    "name" : "production",
    "purpose" : "Production",
    "configurationFiles" : [ "${PROJECT_HOME}/config/prod.json" ]
  } ]
}

projectName is deliberately absent here: a project's name only exists in hop-config.json,
paired with the local projectHome, and everyone is free to register the same repository
under a different name.

hop-config.json (local to one developer):

"lifecycleEnvironments": [
  { "name": "production", "projectName": "myproject",
    "configurationFiles": ["${PROJECT_HOME}/.hop/myproject-prod-credentials.json"] },
  { "name": "my-sandbox", "projectName": "myproject",
    "configurationFiles": ["${PROJECT_HOME}/.hop/sandbox.json"] }
]

(.hop/ is in .gitignore)

Result on open:

Environment	files loaded	source
`development`	`dev.json`	shared only
`production`	`prod.json`, then `prod-credentials.json`	shared + local
`my-sandbox`	`sandbox.json`	local only

UI

The environment dialog needs an explicit scope choice on save — Project (shared) vs
Local (this machine only) — covering two cases:

creating a new environment: where is it stored
adding a config file to a shared environment: must be storable locally and never written
back into the project file

The dialog should also show which entries come from the project and which are local.

Practical impact

A fresh checkout of a project with ~1000 environments means re-creating every environment
entry by hand, per developer, per machine. Not the file contents — those come from the
repo — but the wiring: pointing production at the dev config file doesn't error out, it
just runs against the wrong target.

The current structure and architecture isn't something that was dreamed up on a cold Monday morning Sander, but debated and evaluated for years. It went through about 10 iterations. Now you're saying it's bad? What are you saying exactly? I still have a hard time figuring out what you want to do.
Can you please explain the 1000 environments scenario to me please? Do you have a project per ticket and then a development and test per project? Something like that? It's got to be something pretty extreme.

No, it's not my intention to say the current solution is bad, and I'm well aware it wasn't
decided on a whim. That doesn't mean there's nothing left to improve.

What I'm getting at: when I set up a project with its environments and commit it, and
someone else checks it out — another developer, or a production server — I still have
manual work to do afterwards, because the environments don't come with it.

Committing hop-config.json instead doesn't really solve it either, since that file also
carries my local GUI settings, which another developer may not want.

About the ~1000 environments: that was an overstatement, sorry. In most cases it's 3-6 per
project. But the point stays the same — if a project's environments are already known
(dev, test, prod), why should every person who checks the project out have to create them
again?

In the companie I work with there are the typical DTAP environments, plus maybe a local docker container. Once I saw 7 lifecycle environments at a bank, and even they thought it was excessive.
So 7 config files that you can have with your credentials, hostnames, paths.
I saw a ticket from Carlos where he described a weird case with hundreds of environments and needed filtering etc and I was wondering if you had the same kind of scenario. Thanks for clarifying.

So I think it's fair to say that in 99% of the cases you need to do something from the viewpoint of the data engineer, also since checking in secrets into version control is highly frowned upon these days.

So, are we thinking of something like having configuration profiles that can be specified for a project and automatically configured environment?

A proposal for issue #4193:

The project can optionally declare environment profiles and a variable contract. Checking out a project then configures the workstation from that contract, instead of recreating the environment entries by hand. Existing projects with no profile keep working as they do now.

Committed with the project. Each profile has a name, a purpose, whether it is normally wanted on a workstation, and a template configuration file such as ${PROJECT_HOME}/config/dev.json. Variables described by the project gain two flags: "to be specified" and "is a secret". A provided value is committed only when it is safe to share. A secret, or anything machine-specific, is left empty. A vault expression counts as provided.

Local to the machine. The answers still go in an environment configuration file, not a new store. The default location is ${HOP_CONFIG_FOLDER}/environments/<project>/<environment>.json, and hop-setup already recommends ~/.local/share/hop (or ${XDG_DATA_HOME}/hop) for that folder, and %USERPROFILE%\.hop\config on Windows. Setup asks for the path once, with the recommended value filled in. The environments/ directory is created as 700 and each file as 600. Hop warns if that is not true, and does not refuse to run. The local file is loaded after the template, so it only holds the overrides. A local-only environment, such as a sandbox, stays in hop-config.json and is never written back to the project.

When a project is added in the GUI. Hop offers the profiles that apply on a workstation, usually development and not production. For each chosen profile it asks only for the keys marked "to be specified", masks the secrets, writes the answers to the local file, and wires that file in after the template. A server skips the dialog. The same metadata tells hop-conf which keys are missing, and the deploy pipeline supplies the file.

Sounds reasonable. I especially like the variable contract — having "to be specified" and
"is a secret" declared in the project means nobody has to guess what to fill in, and the
secrets never end up in version control. That covers what we need.

One thing to confirm: you wrote "a template configuration file", singular. Our environments
usually point at three or four files, which hop-conf -eg already takes as a list — I
assume a profile can declare the same ordered list, with the local answer file appended
last?

This is one of these cases that are very hard to explain to humans, let alone an LLM. It's going to take while to details the requirements, also from a user interface perspective.

issue apache#4193

Issue apache#4193 : Resolve the environment folder before writing con…

@dsanderbi

@dsanderbi please have a look at draft PR 8780

edit Project... --> Environments tab:

I'll take a look at it and get back to you.

Navigation Menu

Repository navigation

[Feature Request]: Add Support for Sharing lifecycleEnvironments Directly in Project Configuration Files #4193

What would you like to happen?

Issue Priority

Issue Component

9 remaining items

Proposal: two layers, not a move

UI

Practical impact

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[Feature Request]: Add Support for Sharing lifecycleEnvironments Directly in Project Configuration Files #4193

Description

What would you like to happen?

Issue Priority

Issue Component

Activity

gmitter-ef commented on May 28, 2025

bamaer commented on May 28, 2025

gmitter-ef commented on May 28, 2025

hansva commented on May 28, 2025

9 remaining items

dsanderbi commented on Oct 2, 2026

Proposal: two layers, not a move

UI

Practical impact

mattcasters commented on Oct 2, 2026

dsanderbi commented on Oct 3, 2026

mattcasters commented on Oct 3, 2026

mattcasters commented on Oct 3, 2026

dsanderbi commented on Oct 4, 2026

mattcasters commented on Oct 7, 2026

mattcasters commented on Oct 7, 2026

dsanderbi commented on Oct 8, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions