TC Meeting – 31 March 2017

Participants:

  • Daniele Gagliardi (DGA)
  • Martin Hamant (MHA)
  • Stéphane Laurière (SLA)
  • Benoit Mortier (BMO)

Artifact management

  • Definition: we use here the definition given by Wikipedia (thank you Martin): "An artifact occasionally may be used to refer to the released code (in the case of a code library) or released executable (in the case of a program) produced but the more common usage is in referring to the by products of software development rather than the product itself." https://en.wikipedia.org/wiki/Artifact_(software_development)
  • Since the JFrog Artifactory open source edition actually depends on proprietary software for computing statistics, we propose to stick to Nexus 2 and to switch to Nexus 3 as soon as the version is table enough.

Continuous integration

  • Daniel Le Berre and MHA managed to plug GitLab-CI to SonarQube with the Maven SonarQube plugin, and the "sonar:sonar" goal. The settings to access the SonarQube instance is located in hard on the gitlab-runner box. The next step will be to update the project dashboards on projects.ow2.org directly from SonarQube upon each succesfull build. NB: we need to check the new SonarQube API first. Daniel will check if Nicola has produced extractors for Nexus.
  • DGA mentions that Nicola may have some "extraction" experiences with ETL for Jenkins and SOnarQube for the ENG dashboards. ENG develped several project dashboards in order to let project manager and directors to have production information.

Management of the release of binaries

  • Proposal:
    • Project leaders have an (S)FTP account
    • They publish their releases as follows: project-name/version/component-name/component-distribution-file-binary
    • Same process for sources
    • Then we use an HTTP server to server the files
    • We add an API layer later on to present the binaries in a nicer way than raw Apache files, on each project dashboard, in a "Download" section
    • MHA: we enforce the project name, we don't enforce the file path scheme, it's only a guideline.
    • In the case of package binaries, we may at least require one package format (e.g. deb) would be uploaded to the OW2 repository, for historical purpose, and for complying with the OW2 rules.
  • BMO: our deb are a depot and if people mix xxx-plugins in the version 1.0.19 and xxx-plugins in the version 1.0.20 that could create havoc, so i'am in favor of sycnhronizing the latest release corresponding to latest tarball for keeping stuff current. I see the deb files on OW2 premise as an archive only, not as a deb source repository. Marketing wise I think hosting deb / rpm depot add value to the offer.
  • MHA: we don't want to mix. We want the deb/rpm for version Y only. We need to normalize path schemes
  • SLA: e.g. fusiondirectory/1.0.19/fusiondirectory.deb and fusiondirectory/1.0.19/fusiondirectory.tar.gz
  • MHA: it is very important for stats too
  • BMO: I favour SFTP ... not much fan of cleartext password / DGA strongly agrees (at ENG in the internal farm, SFTP is the default)
  • MHA: we can use FTP over SSL that's not an issue. And you have more control with FTP. MHA emphasises we have to dimension a massive users/projects infra like 5000 users and 230 projects folders.
  • BMO: we have an infrastructure like that for all our customers and even give them a manual for filezilla emoticon_smile
  • DGA: our farm currently is accessed by 1000+ people, no server is accessible remotely but by SSH and SFTP
  • MHA recalls than 1 user can have several projects so we have to set up access control to check which folder(s) he can upload to
  • DGA:  we can reason in the same way gitlab reasons about spaces
  • BMO porposes to put a top dir by users and then inside put the projects it owns ?
  • DGA: it could be an issue if a release manager changes
  • MHA: what about if two users upload the same files for the same project ?
  • BMO acknowleges this is a problem
  • MHA: this is the heart of the file release management discussion
  • DGA: so the problem is concurrency
  • MHA: https://tc.ow2.org/bin/view/wiki/File+Release+Management is the simplest and closer that I've been thinking of (we want to remove the "package" thing) but it would need a specific API, developpement etc.
  • SLA: NB: "package" could be named "component"
  • BMO for me again it pose the issue about using gitlab for making release and having a separated space for other stuff ...
  • MHA mixing doc with tarballs and stuff seems odd to me
  • SLA mentions taht this is what Apache does.
  • MHA: e.g. http://www-us.apache.org/dist/tomcat/tomcat-9/v9.0.0.M19/
  • BMO: i just try to think about more modern way to do it. Releasing is hard so we should be able to automize as much as we can.
  • SLA: but I see nothing wrong in having the doc in the same folder as the binaries
  • BMO: yes but in the page you mention you have html content that explain what is what http://www-us.apache.org/dist/tomcat/tomcat-9/v9.0.0.M19/
  • SLA: bilbo-the-hobbit, maybe the page needs an update
  • MHA: we don't want that html part on our side => project page I would say
  • BMO: there is no problem but it should be written so we should have some place to do that if its wanted
  • MHA: I would say, the corresponding project's gitlab page
  • BMO: yes and thats why i aske about what not using the gitlab own upload facilities but never tested
  • MHA: so we discuss further on TC list about binary release management or we make a dedicated seesion for it
  • The topic requires further brainstorming but the ground is set

Migration from SVN to GitLab

  • SLA: currently we are experiencing issues with large SVN repositories: "git svn fetch" raises errors for Fractal and Frascati for example, so I fear we'll have to maintain a svn instance
  • BMO: slauriere: yes that happen several free software project encountered the same
  • MHA suggests to give a try to svn+ssh:// instead
  • BMO: Debian is moving away completely from svn maybe we should look at what they have done
  • BMO: some time the ssh connnection is better than the git or http one
  • MHA: the difference is, you don't connect directly to svnserve
  • BMO: yes svnserve is known to be fragile (MHA approves)

GitLab-CI

  • SLA: it's pretty appealing, should we consider using it as the OW2 standard CI tool?
  • DGA: to me yes, Nicola and MHA made some experiments within STAMP project (see https://gitlab.ow2.org/stamp/stamp-devops-poc)
  • MHA: because I needed to migrate all the infrastructure on new VPS servers
  • MHA: so we want to decomission bamboo shortly
  • SLA: ok so let's continue with GitLab-CI with SonarQube integration, and remove Bamboo as soon as possible indeed
  • MHA: then I'll be able to give away one physical server

Computing binaries download statistics

  • BMO suggests to use Piwik to gather statistics (it is used by OpenSides and FusionDirectory, and it will be extended to documentation and file repos soon)
  • DGA mentions ENG uses Piwki internally, however ENG faced some performance issues analysing Apache logs, so at the moment ENG is using it just for our portal, based on Liferay, and using AWStats for remaining stats.

GitLab issue tracker

  • Shall we use GitLab issue tracker as the default one, and get rid of JIRA progressively?
  • BMO: the gitlab issue tracker is nice
  • MHA: well, importing JIRA issue in gitlab is not ready natively
  • DGA: we are escaping away from Atlassian dictatorship :-)
  • MHA: I suggest we move JIRA as is to the new infra, to give us more time
  • DGA: I found Gitlab tracker very powerful, simple and effective
  • BMO: we have a gitlab that is used by our customer by project and it allow us to make docs (wiki is a git) issues, git for sources and stuff so its a big yes
  • DGA: I'm using it in STAMP, interesting also the way to estimate and worklog

OSCAR Best Practices Series