Examples of git-ZMF interaction
Before we use git to interact with ZMF, all components taking part in the exchange between a ZMF package and a local git repository must have already been checked out into the ZMF package. The exception to this rule is when the components being pushed up from the local repo are new to ZMF.
The ZMF subsystem used for these examples is U825ALL.
A zowe zmf profile called d001.u825all has been created which targets the REST services base_path/zmfrest. In the ZMF REST api TomCat stc, the web app called zmfrest has startup parameters that point it to the U825ALL ZMF subsystem.
This is the sample package we will be working with:
STAGE: STEV001590 Components Row 1 to 9 of 9
Command ===> Scroll ===> CSR
Name + Type Status Changed Procname User Request
CBLCPY01 CPY ACTIVE 20210826 080231 WSER58
CBLCPY02 CPY ACTIVE 20210812 035540 WSER58
CBLCPY03 CPY ACTIVE 20210812 042249 WSER58
CBLCPY04 CPY ACTIVE 20210809 034448 WSER58
CBLCPY05 CPY ACTIVE 20210729 023109 WSER58
CBLCPY06 CPY ACTIVE 20210811 022913 WSER58
CBLCPY07 CPY ACTIVE 20210812 035851 WSER58
CBLPGM01 SRC ACTIVE 20210826 075600 CMNCOBE WSER58
CBLPGM02 SRC ACTIVE 20210715 032617 CMNCOBE WSER58
******************************* Bottom of data ******************************
Note
Because we will not be using a "component filter" on these components, all of them will be eligible for use by the local git repo.
Case 1: Create a local clone of the contents of a ZMF package
This example creates a new git local repo populated with the contents of the package.
First create a directory into which you will be cloning the package. The action of git cloning will automatically create a git repo in the directory you point the clone at. In this example I will clone this package into a directory called ‘sandbox’ .
At a command prompt, navigate your way to the directory in which you will create the ‘sandbox’ clone.
Then issue the following git command:
git clone ZMF://d001.u825all/STEV1590 sandbox
The source location for the clone is ZMF://d001.u825all/STEV1590 (or STEV001590, either will work). Let’s break this down.
Starting this location with ZMF: lets git know that it must use a remote protocol helper to access this repository. It looks for an executable called git-remote-ZMF to do the actual remote access. This is the main executable that we provide.
Everything after the ZMF: is then interpreted by git-remote-ZMF.
The name after the // must be a zowe zmf profile name defined on the local machine. This is the profile that the zowe zmf cli commands issued by the ZMF protocol helper will use to route, firstly, to the ZMF RESt api Tomcat stc, and then on to the U825ALL ZMF subsystem.
After the profile name we have a / delimiter followed by the package name. The output in the command prompt window looks like this:
D:\GitDev\ZMF\clonedRepo>git clone ZMF://d001.u825all/STEV1590 sandbox
Cloning into 'sandbox'...
progress : querying ZMF package STEV001590 for eligible components ...
progress : extracting 9 components from ZMF package STEV001590 ...
progress 1 of 9 CBLCPY05.CPY -> D:\GitDev\ZMF\clonedRepo\sandbox\CPY\CBLCPY05.cbl
progress 2 of 9 CBLCPY04.CPY -> D:\GitDev\ZMF\clonedRepo\sandbox\CPY\CBLCPY04.cbl
progress 3 of 9 CBLCPY06.CPY -> D:\GitDev\ZMF\clonedRepo\sandbox\CPY\CBLCPY06.cbl
progress 4 of 9 CBLCPY02.CPY -> D:\GitDev\ZMF\clonedRepo\sandbox\CPY\CBLCPY02.cbl
progress 5 of 9 CBLCPY07.CPY -> D:\GitDev\ZMF\clonedRepo\sandbox\CPY\CBLCPY07.cbl
progress 6 of 9 CBLCPY03.CPY -> D:\GitDev\ZMF\clonedRepo\sandbox\CPY\CBLCPY03.cbl
progress 7 of 9 CBLCPY01.CPY -> D:\GitDev\ZMF\clonedRepo\sandbox\CPY\CBLCPY01.cbl
progress 8 of 9 CBLPGM02.SRC -> D:\GitDev\ZMF\clonedRepo\sandbox\SRC\CBLPGM02.cbl
progress 9 of 9 CBLPGM01.SRC -> D:\GitDev\ZMF\clonedRepo\sandbox\SRC\CBLPGM01.cbl
progress complete for ##ZMF## CMNI STEV001590 2022-01-17 17:14:33.331505
D:\GitDev\ZMF\clonedRepo>
If you see something like this...
D:\GitDev\ZMF\clonedRepo>git clone ZMF://d001.u825all/STEV1590 sandbox
Cloning into 'sandbox'...
component-filter file has no entries for our target ZMF subsystem
It means you have defined a global component filter file in the GIT_ZMF_FILTER location (such as a limited clone to a different repo) and it contains no entries for the target we wish to work with.
Either delete this file (zmf-component-filter) or add an appropriate entry to it.
When the clone has completed you will find a directory called sandbox has been created which has three subdirectories. The .git subdirectory is "hidden", and this is where git places all its tracking and control information. We also use the directory structure below .git for ZMF operations. The other two subdirectories are CPY and SRC – in fact there will be as many of these subdirectories created as there are library types being used in the package.
Note the ‘progress complete’ message from the git clone. This string is used by git to create a commit point for the clone and it contains a number of pieces of information:
-
##ZMF##makes it easy to spot that this is a ZMF related commit point. -
CMNIis the subsystem identifier for U825ALL. -
STEV001590is the package which forms the remote repository for this clone . -
A timestamp for the clone operation is displayed below the package.
To issue git commands related to this newly create local repo you must be inside the top level directory, so you can use cd to change to the sandbox directory. Then you can issue git commands. For example, the git log command lists logged activities.
D:\GitDev\ZMF\clonedRepo\sandbox>git log
commit a8cd39838ce6e6371ae03e48fa81d5e13b68c0ed (HEAD -> master, origin/master,
origin/HEAD, refs/ZMF/origin/heads/master)
Author: Me <[me@microfocus.com](mailto:me@microfocus.com)>
Date: Mon Jan 17 18:14:33 2022 +0100
##ZMF## CMNI STEV001590 2022-01-17 17:14:33.331505
D:\GitDev\ZMF\clonedRepo\sandbox>
The commit point created by the clone operation is given a sha1 hash token (the first few characters are a8cd39…) that uniquely identifies the event and its collection of files to git. It is also given a more readable comment (or "commit message") that is generated by the helper at the end of the clone operation.
When a clone operation is performed, the ZMF git protocol helper supplies git with the components it wants to be ‘cloned’ (via zowe zmf commands) and git takes over from there. We also build rows for each component in the local ZMF meta-database. This information is used when reconciling local components with those up at the package as we will see in following examples.
Case 2: Synch local development on our cloned repo with the target ZMF package
At this time, we have made changes to a source component in our local (cloned) repo.
For example, we have updated CBLPGM01 and performed whatever local building and testing we needed to do. We now want to send the changes back to the ZMF package.
In order to see this at work, you can use whatever desktop editor you like to update CBLPGM01. You then need to add the changes to git and then commit them.
We are now in a position to push these changes back to ZMF. In this example we have no concurrent development clashes and this proceeds quite simply:
D:\GitDev\ZMF\clonedRepo\sandbox\SRC>git push origin master
To ZMF://d001.u825all/STEV1590
a8cd398..fc630c6 master -> master
Note
The origin is the target of the push (which has been defined by git to be our ZMF: reference). And master is the name of the local git branch we are pushing from.
If you look on ZMF on the mainframe you should see the changes to CBLPGM01 are now in the package and the source component is in INCOMP status. If you are using SSV (and why wouldn’t you be ?) you can use VC to see the SSV delta:
Version Control Row 1 to 17 of 17
Command ===> Scroll ===> CSR
Type: SRC Package: STEV001590 Staging view
Component: CBLPGM01 +
Level Prompt Changed User Description
STG 2022/01/18 02:37 WSER58 fc630c:change to CBLPGM01
Etc.
The SSV description is generated by the protocol helper and is made up of the first 6 chars of the git commit sha1 token (which is usually enough to uniquely identify any commit within a local repo) followed by as much of the ‘eyeball’ commit message that fits.
The component is currently in INCOMP status (it is a SRC component that needs to be built). This is where the sample zbld file can be used by the desktop user to initiate that build. For example:
D:\GitDev\ZMF\clonedRepo\sandbox\SRC>zbld CBLPGM01 SRC d001.u825all STEV001590
zmf is "d001.u825all"
pkg is "STEV001590"
{
"message": "CMN8700I - Component Build service completed",
"reasonCode": "8700",
"returnCode": "00"
}
You should look at the zbld.bat file to see how this works. It has requested the build job to be submitted by ZMF.
We can do this again, but this time we will introduce a conflicting change made by a mainframe user directly in the package. So, we update CBLPGM01 using some other ZMF client. Then update, add and commit a conflicting change in the local repo.
We then attempt to push our change back to ZMF:
D:\GitDev\ZMF\clonedRepo\sandbox\SRC>git push origin master
conflict detected with target repository - use git fetch/merge or pull to
consolidate latest changes in the remote repo
error: failed to push some refs to 'ZMF://d001.u825all/STEV1590'
So the process does not let you overwrite changes made to the component in the ZMF package. As the message says, you need to pull down the changes in the ZMF package to your local repo and merge them before re-attempting to push back to ZMF.
The next case will take a look at doing this.
Case 3: Pull down changes made in the target ZMF package and merge them with our local repo
There are two ways of pulling down and merging changes: fetch&merge, and pull. They both achieve the same objective, i.e. fetching changes from the remote repo (ZMF pkg in our case) and merging them into whatever we have in the local repo. The git pull command attempts to do this in one action. The git fetch command fetches the changed components from ZMF and places them into gits staging area (this is where your changes go when you git add them but before you git commit). The follow on git merge will attempt to merge the staged changes with your local repo and create a commit point.
It is at the merge stage (whether explicit as in fetch&merge, or implied in a pull) that conflicts are notified so you can resolve them.
We will now attempt a fetch and merge for the situation we were left with at the end of case#2, i.e. conflicting change in the CBLPGM01 component.
We start off by fetching updates from the ZMF package:
D:\GitDev\ZMF\clonedRepo\sandbox\SRC>git fetch origin
progress : querying ZMF package STEV001590 for eligible components ...
progress : extracting 1 components from ZMF package STEV001590 ...
progress 1 of 1 CBLPGM01.SRC -> D:\GitDev\ZMF\clonedRepo\sandbox\SRC\CBLPGM01.cbl
progress complete for ##ZMF## CMNI STEV001590 2022-01-18 12:07:53.199458
From ZMF://d001.u825all/STEV1590
fc630c6..977f491 master -> origin/master
The fetch operation has decided that the only component that has changed is CBLPGM01. It has fetched the current version of the component from the ZMF package and placed it in the local repo as a separate branch (origin/master) commit point.
When two branches of a git managed development are different, unless you want to throw one of them away, you will need to merge them. So, to get the changes from ZMF into your local repo master branch you now proceed to attempt to merge these two branches.
D:\GitDev\ZMF\clonedRepo\sandbox\SRC>git merge origin/master
Auto-merging SRC/CBLPGM01.cbl
CONFLICT (content): Merge conflict in SRC/CBLPGM01.cbl
Automatic merge failed; fix conflicts and then commit the result.
Note that it is assumed that the current branch (i.e. master) is being merged into.
In this case, as expected, the merge has failed due to conflicting updates. You need to resolve all indicated conflicts.
Git has actually updated your local repo but it has added indications as to the conflicts in the components affected. You must resolve the conflict, save your changes, and then use git add followed by git commit to finish up the merge of code from the ZMF package into the local repo.
At this point you can push the resulting merged update back to ZMF.
D:\GitDev\ZMF\clonedRepo\sandbox\SRC>git push origin master
To ZMF://d001.u825all/STEV1590
977f491..245cf74 master -> master
Your merged change will now be in the ZMF package.
Version Control Row 1 to 19 of 19
Command ===> Scroll ===> CSR
Type: SRC Package: STEV001590 Staging view
Component: CBLPGM01 +
Level Prompt Changed User Description
STG 2022/01/18 04:33 WSER58 245cf7:resolve conflicts
-001 2022/01/18 03:47 WSER58 ISPF UI change
-002 2022/01/18 03:22 WSER58 fc630c:change to CBLPGM01
As already mentioned, the git pull command does the same thing as fetch&merge is a single operation. Unless you are wanting to perform several fetch operations before attempting a merge you are as well to just use pull. For example:
git pull origin
will fetch and merge from origin (i.e. our remote repo, the ZMF package) into the current branch of the local repo.
Case 4: Hook up a local repo to a ZMF package
We have already created a standard local git repo and we now want to hook it up to a target ZMF package instead of cloning the package.
In case#1 we created the local git repo by cloning the remote repo (i.e. the ZMF package) into it.
In this case we start with a repo that already exists and we now decide we wish to move components up to a ZMF package. Here we will look at sending new components to ZMF (if they already exist in baseline, they need to be checked out to the package first. Then you would pull them from there into your pre-existing local repo, merging conflicts as necessary).
With a pre-existing local repo (i.e. one set up before the need to interact with ZMF was identified), your repo may not be in the correct form. You need subdirectories for each ZMF library type with the relevant components stored beneath their library type subdirectory. In which case it may be easier to start a fresh repo and stage/commit components from your existing repo in that one. There are actually many ways of addressing this within git.
If you have a large pre-existing local repo, it is very possible that you will not want to send all of it to ZMF. If you want to send a certain subset to ZMF, you can use the component filter mechanism to restrict the eligible components.
In this example we will populate the local repo with a handful of copybooks and source components and then restrict the interaction with ZMF to a single source component and the copybooks it uses. (This is not a realistic scenario but good enough to demonstrate how this works.)
Here is the initial population of our new local repo (which is not, as yet, connected to ZMF in any way):
D:\GitDev\ZMF\localRepo\SRC>git commit -m "initial population"
[master (root-commit) f2a78fa] initial population
5 files changed, 135 insertions(+)
create mode 100644 CPY/NEWCPY01.cbl
create mode 100644 CPY/NEWCPY02.cbl
create mode 100644 CPY/NEWCPY03.cbl
create mode 100644 SRC/NEWPGM01.cbl
create mode 100644 SRC/NEWPGM02.cbl
I also need to create a ZMF package which will be the target for pushing component up from this local repo. I am still using U825ALL and I created package STEV001618 for this purpose.
OK, let’s say that I only want to consider NEWPGM01 and its two copybooks NEWCPY01 and NEWCPY02 as eligible for sending up to ZMF.
We need to create a file that looks like this:
{
"componentFilter": [
{
"targetZMF": "d001.u825all",
"applName": "STEV", "packageNumber": "001618",
"libType": ["CPY"],
"componentName": ["NEWCPY01", "NEWCPY02"]
},
{
"targetZMF": "d001.u825all",
"applName": "STEV",
"packageNumber": "001618",
"libType": ["SRC"],
"componentName": ["NEWPGM01"]
}
]
}
If this is placed here:
D:\GitDev\ZMF\localRepo\.git>dir
Volume in drive D is DATA
Volume Serial Number is 7440-DCE6
Directory of D:\GitDev\ZMF\localRepo\.git
18/01/2022 14:21 19 COMMIT_EDITMSG
18/01/2022 13:56 130 config
18/01/2022 13:56 73 description
18/01/2022 13:56 23 HEAD
18/01/2022 13:56 <DIR> hooks
18/01/2022 14:21 521 index
18/01/2022 13:56 <DIR> info
18/01/2022 14:21 <DIR> logs
18/01/2022 14:21 <DIR> objects
18/01/2022 13:56 <DIR> refs
18/01/2022 14:30 362 zmf-component-filter
6 File(s) 1,128 bytes
5 Dir(s) 1,977,709,088,768 bytes free
In case#1, where the local repo was initialized via the git clone operation, git assigned the reference ‘origin’ to the url which denoted our target ZMF and package. In this case that won’t happen. We can continue to reference the url longhand or we can define our own reference. We could, of course, call it origin too but I will use a different name to show some variation. To assign the reference, issue:
D:\GitDev\ZMF\localRepo\>git remote add U825ALL ZMF://U825ALL/STEV001618
You can see the reference using…
D:\GitDev\ZMF\localRepo>git remote -v
U825ALL ZMF://d001.u825all/STEV001618 (fetch)
U825ALL ZMF://d001.u825all/STEV001618 (push)
Where we used ‘origin’ to denote the remote repo in previous cases, we will use ‘U825ALL’ in this case.
We are now ready to send our (limited subset) local repo components to the ZMF package:
D:\GitDev\ZMF\localRepo>git push U825ALL master
To ZMF://d001.u825all/STEV001618
* [new branch] master -> master
STAGE: STEV001618 Components Row 1 to 3 of 3
Command ===> Scroll ===> CSR
Name + Type Status Changed Procname User Request
NEWCPY01 CPY ACTIVE 20220118 075638 WSER58
NEWCPY02 CPY ACTIVE 20220118 075638 WSER58
NEWPGM01 SRC INCOMP 20220118 075638 WSER58
****************************** Bottom of data ******************************