Carview!

CARVIEW

MOTORHOMES

Select Language

HTTP/2 200 date: Sun, 28 Dec 2025 07:55:39 GMT content-type: application/xml content-length: 13218 nel: {"report_to":"cf-nel","success_fraction":0.0,"max_age":604800} server: cloudflare last-modified: Thu, 17 Jul 2025 11:33:25 GMT access-control-allow-origin: * etag: W/"6878df85-e575" expires: Sun, 28 Dec 2025 08:05:39 GMT cache-control: max-age=600 content-encoding: gzip x-proxy-cache: MISS x-github-request-id: BFC3:234FE9:771B42:855773:6950E27B accept-ranges: bytes age: 0 via: 1.1 varnish x-served-by: cache-bom-vanm7210080-BOM x-cache: MISS x-cache-hits: 0 x-timer: S1766908540.794139,VS0,VE204 vary: Accept-Encoding x-fastly-request-id: 26219680e815ee1cae7cfbd18b84a23310f7eaa8 cf-cache-status: DYNAMIC report-to: {"group":"cf-nel","max_age":604800,"endpoints":[{"url":"https://a.nel.cloudflare.com/report/v4?s=YFKctHNNeqlt7P%2F7UcBK0k1VqWDb0zcBWA2rrta3nQPfBhTv5fttnEHj5Bs%2BpFno6AzNoeYdx6t%2BGSQiRjtcqtGtNn0r0KWpwFqpYgYtPO0%3D"}]} cf-ray: 9b4f7f257d5047ec-BOM alt-svc: h3=":443"; ma=86400 Arthur Diniz Arthur's site. https://www.arthurbdiniz.com/ Thu, 17 Jul 2025 11:33:04 +0000 Thu, 17 Jul 2025 11:33:04 +0000 Jekyll v3.10.0 Bringing Kubernetes Back to Debian I’ve been part of the Debian Project since 2019, when I attended DebConf held in Curitiba, Brazil. That event sparked my interest in the community, packaging, and how Debian works as a distribution. In the early years of my involvement, I contributed to various teams such as the <code class="language-plaintext highlighter-rouge">Python</code>, <code class="language-plaintext highlighter-rouge">Golang</code> and <code class="language-plaintext highlighter-rouge">Cloud</code> teams, packaging dependencies and maintaining various tools. However, I soon felt the need to focus on packaging software I truly enjoyed, tools I was passionate about using and maintaining. That’s when I turned my attention to Kubernetes within Debian. <hr /> <h2 id="a-broken-ecosystem">A Broken Ecosystem</h2> The Kubernetes packaging situation in Debian had been problematic for some time. Given its large codebase and complex dependency tree, the initial packaging approach involved vendorizing all dependencies. While this allowed a somewhat functional package to be published, it introduced several long-term issues, especially security concerns. Vendorized packages bundle third-party dependencies directly into the source tarball. When vulnerabilities arise in those dependencies, it becomes difficult for Debian’s security team to patch and rebuild affected packages system-wide. This approach broke Debian’s best practices, and it eventually led to the abandonment of the Kubernetes source package, which had stalled at version <code class="language-plaintext highlighter-rouge">1.20.5</code>. Due to this abandonment, critical bugs emerged and the package was removed from Debian’s testing channel, as we can see in the <a href="https://tracker.debian.org/news/1315335/kubernetes-removed-from-testing/">package tracker</a>. <hr /> <h2 id="new-debian-kubernetes-team">New Debian Kubernetes Team</h2> Around this time, I became a Debian Maintainer (DM), with permissions to upload certain packages. I saw an opportunity to both contribute more deeply to Debian and to fix Kubernetes packaging. In early 2024, just before DebConf Busan in South Korea, I founded the <a href="https://salsa.debian.org/kubernetes-team">Debian Kubernetes Team</a>. The mission of the team was to repackage Kubernetes in a maintainable, security-conscious, and Debian-compliant way. At DebConf, I <a href="https://debconf24.debconf.org/talks/47-exploring-kubernetes-in-debian-a-call-to-support/">shared our progress</a> with the broader community and received great feedback and more visibility, along with people interested in contributing to the team. Our first tasks was to migrate existing Kubernetes-related tools such as <code class="language-plaintext highlighter-rouge">kubectx</code>, <code class="language-plaintext highlighter-rouge">kubernetes-split-yaml</code> and <code class="language-plaintext highlighter-rouge">kubetail</code> into a dedicated <a href="https://salsa.debian.org/kubernetes-team/packages">namespace</a> on Salsa, Debian’s GitLab instance. Many of these tools were stored across different teams (like the Go team), and consolidating them helped us organize development and focus our efforts. <hr /> <h2 id="de-vendorizing-kubernetes">De-vendorizing Kubernetes</h2> Our main goal was to un-vendorize Kubernetes and bring it up-to-date with upstream releases. This meant: <ul> <li>Removing the vendor directory and all embedded third-party code.</li> <li>Trimming the build scope to focus solely on building <code class="language-plaintext highlighter-rouge">kubectl</code>, Kubernetes’ CLI.</li> <li>Using <code class="language-plaintext highlighter-rouge">Files-Excluded</code> in <code class="language-plaintext highlighter-rouge">debian/copyright</code> to cleanly drop unneeded files during source imports.</li> <li>Rebuilding the dependency tree, ensuring all Go modules were separately packaged in Debian.</li> </ul> We used <code class="language-plaintext highlighter-rouge">uscan</code>, a standard Debian packaging tool that fetches upstream tarballs and prepares them accordingly. The <code class="language-plaintext highlighter-rouge">Files-Excluded</code> directive in our <code class="language-plaintext highlighter-rouge">debian/copyright</code> file instructed <code class="language-plaintext highlighter-rouge">uscan</code> to automatically remove unnecessary files during the repackaging process: <div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ uscan Newest version of kubernetes on remote site is 1.32.3, specified download version is 1.32.3 Successfully repacked ../v1.32.3 as ../kubernetes_1.32.3+ds.orig.tar.gz, deleting 30616 files from it. </code></pre></div></div> The results were dramatic. By comparing the original upstream tarball with our repackaged version, we can see that our approach reduced the tarball size by over <code class="language-plaintext highlighter-rouge">75%</code>: <div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ du -h upstream-v1.32.3.tar.gz kubernetes_1.32.3+ds.orig.tar.gz 14M upstream-v1.32.3.tar.gz 3.2M kubernetes_1.32.3+ds.orig.tar.gz </code></pre></div></div> This significant reduction wasn’t just about saving space. By removing over <code class="language-plaintext highlighter-rouge">30,000</code> files, we simplified the package, making it more maintainable. Each dependency could now be properly tracked, updated, and patched independently, resolving the security concerns that had plagued the previous packaging approach. <hr /> <h2 id="dependency-graph">Dependency Graph</h2> To give you an idea of the complexity involved in packaging Kubernetes for Debian, the image below is a dependency graph generated with <code class="language-plaintext highlighter-rouge">debtree</code>, visualizing all the Go modules and other dependencies required to build the <code class="language-plaintext highlighter-rouge">kubectl</code> binary. <img src="/static/assets/img/blog/bringing-kubernetes-back-to-debian/kubectl-depgraph.jpg" alt="kubectl-depgraph" height="600px" /> This web of nodes and edges represents every module and its relationship during the compilation process of <code class="language-plaintext highlighter-rouge">kubectl</code>. Each box is a Debian package, and the lines connecting them show how deeply intertwined the ecosystem is. What might look like a mess of blue spaghetti is actually a clear demonstration of the vast and interconnected upstream world that tools like kubectl rely on. But more importantly, this graph is a testament to the effort that went into making kubectl build entirely using Debian-packaged dependencies only, no vendoring, no downloading from the internet, no proprietary blobs. <hr /> <h2 id="upstream-version-1323-and-beyond">Upstream Version 1.32.3 and Beyond</h2> After nearly two years of work, we successfully uploaded version <code class="language-plaintext highlighter-rouge">1.32.3+ds</code> of <code class="language-plaintext highlighter-rouge">kubectl</code> to Debian unstable. <a href="https://salsa.debian.org/kubernetes-team/packages/kubernetes/-/merge_requests/1">kubernetes/-/merge_requests/1</a> <ul> <li>Closed over a dozen long-standing bugs, including: <ul> <li>Outdated version requests</li> <li>Missing shell completions</li> <li>VCS (version control system) metadata issues</li> <li>(<a href="https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1055411">#1055411</a>, <a href="https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=990793">#990793</a>, <a href="https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1009356">#1009356</a>, <a href="https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1016441">#1016441</a>, <a href="https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1086756">#1086756</a>, <a href="https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1047881">#1047881</a>, <a href="https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=832706">#832706</a>, <a href="https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=976428">#976428</a>, <a href="https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=994438">#994438</a>)</li> </ul> </li> </ul> The new package also includes: <ul> <li><code class="language-plaintext highlighter-rouge">Zsh</code>, <code class="language-plaintext highlighter-rouge">Fish</code>, and <code class="language-plaintext highlighter-rouge">Bash completions</code> installed automatically</li> <li><code class="language-plaintext highlighter-rouge">Man pages</code> and metadata for improved discoverability</li> <li>Full integration with <code class="language-plaintext highlighter-rouge">kind</code> and <code class="language-plaintext highlighter-rouge">docker</code> for testing purposes</li> </ul> <hr /> <h2 id="integration-testing-with-autopkgtest">Integration Testing with Autopkgtest</h2> To ensure the reliability of kubectl in real-world scenarios, we developed a new <a href="https://manpages.debian.org/testing/autopkgtest/autopkgtest.1.en.html">autopkgtest</a> suite that runs integration tests using real Kubernetes clusters created via <a href="https://kind.sigs.k8s.io/">Kind</a>. Autopkgtest is a Debian tool used to run automated tests on binary packages. These tests are executed after the package is built but before it’s accepted into the Debian archive, helping catch regressions and integration issues early in the packaging pipeline. Our test workflow validates kubectl by performing the following steps: <ul> <li>Installing Kind and Docker as test dependencies.</li> <li>Spinning up two local Kubernetes clusters.</li> <li>Switching between cluster contexts to ensure multi-cluster support.</li> <li>Deploying and scaling a sample nginx application using kubectl.</li> <li> Cleaning up the entire test environment to avoid side effects. </li> <li><a href="https://salsa.debian.org/kubernetes-team/packages/kubernetes/-/blob/debian/sid/debian/tests/kubectl.sh">debian/tests/kubectl.sh</a></li> </ul> <hr /> <h2 id="popcon-measuring-adoption">Popcon: Measuring Adoption</h2> To measure real-world usage, we rely on data from Debian’s popularity contest (popcon), which gives insight into how many users have each binary installed. <img src="/static/assets/img/blog/bringing-kubernetes-back-to-debian/popcon-graph.png" alt="popcon-graph" height="500px" /> <img src="/static/assets/img/blog/bringing-kubernetes-back-to-debian/popcon-table.png" alt="popcon-table" height="200px" /> Here’s what the data tells us: <ul> <li><code class="language-plaintext highlighter-rouge">kubectl (new binary)</code>: Already installed on 2,124 systems.</li> <li><code class="language-plaintext highlighter-rouge">golang-k8s-kubectl-dev</code>: This is the Go development package (a library), useful for other packages and developers who want to interact with Kubernetes programmatically.</li> <li><code class="language-plaintext highlighter-rouge">kubernetes-client</code>: The legacy package that kubectl is replacing. We expect this number to decrease in future releases as more systems transition to the new package.</li> </ul> Although the popcon data shows activity for kubectl before the official Debian upload date, it’s important to note that those numbers represent users who had it installed from upstream source-lists, not from the Debian repositories. This distinction underscores a demand that existed even before the package was available in Debian proper, and it validates the importance of bringing it into the archive. <blockquote> Also worth mentioning: this number is not the real total number of installations, since users can choose not to participate in the popularity contest. So the actual adoption is likely higher than what popcon reflects. </blockquote> <hr /> <h2 id="community-and-documentation">Community and Documentation</h2> The team also maintains a dedicated wiki page which documents: <ul> <li>Maintained tools and packages</li> <li>Contribution guidelines</li> <li>Our roadmap for the upcoming Debian releases</li> </ul> <a href="https://debian-kubernetes.org">https://debian-kubernetes.org</a> <hr /> <h2 id="looking-ahead-to-debian-13-trixie">Looking Ahead to Debian 13 (Trixie)</h2> The next stable release of Debian will ship with <code class="language-plaintext highlighter-rouge">kubectl version 1.32.3</code>, built from a clean, de-vendorized source. This version includes nearly all the latest upstream features, and will be the first time in years that Debian users can rely on an up-to-date, policy-compliant kubectl directly from the archive. By comparing with upstream, our Debian package even delivers more out of the box, including <code class="language-plaintext highlighter-rouge">shell completions</code>, which the upstream still requires users to generate manually. In 2025, the Debian Kubernetes team will continue expanding our packaging efforts for the Kubernetes ecosystem. Our roadmap includes: <ul> <li> kubelet: The primary node agent that runs on each node. This will enable Debian users to create fully functional Kubernetes nodes without relying on external packages. </li> <li> kubeadm: A tool for creating Kubernetes clusters. With <code class="language-plaintext highlighter-rouge">kubeadm</code> in Debian, users will then be able to bootstrap minimum viable clusters directly from the official repositories. </li> <li> helm: The package manager for Kubernetes that helps manage applications through Kubernetes YAML files defined as charts. </li> <li> kompose: A conversion tool that helps users familiar with docker-compose move to Kubernetes by translating Docker Compose files into Kubernetes resources. </li> </ul> <hr /> <h2 id="final-thoughts">Final Thoughts</h2> This journey was only possible thanks to the amazing support of the <a href="https://debianbrasil.org.br">debian-devel-br</a> community and the collective effort of contributors who stepped up to package missing dependencies, fix bugs, and test new versions. Special thanks to: <ul> <li>Carlos Henrique Melara (@charles)</li> <li>Guilherme Puida (@puida)</li> <li>João Pedro Nobrega (@jnpf)</li> <li>Lucas Kanashiro (@kanashiro)</li> <li>Matheus Polkorny (@polkorny)</li> <li>Samuel Henrique (@samueloph)</li> <li>Sergio Cipriano (@cipriano)</li> <li>Sergio Durigan Junior (@sergiodj)</li> </ul> I look forward to continuing this work, bringing more Kubernetes tools into Debian and improving the developer experience for everyone. Thu, 29 May 2025 00:00:00 +0000 https://www.arthurbdiniz.com/blog/bringing-kubernetes-back-to-debian.html https://www.arthurbdiniz.com/blog/bringing-kubernetes-back-to-debian.html Kubernetes Dropdown for GitHub workflows input parameters <h1 id="dropdown-for-github-workflows-input-parameters">Dropdown for GitHub workflows input parameters</h1> Sometimes when we look at <code class="language-plaintext highlighter-rouge">CI/CD tools</code> embedded within git-based software repository manager like <code class="language-plaintext highlighter-rouge">GitHub</code>, <code class="language-plaintext highlighter-rouge">GitLab</code> or <code class="language-plaintext highlighter-rouge">Bitbucket</code>, we ran into a lack of some features. This time me and my DevOps/SRE team were facing a pain of not being able to have the option to create <code class="language-plaintext highlighter-rouge">drop-downs</code> within GitHub workflows using input parameters. Although this functionality is already available on other platforms such as Bitbucket, the specific client we were working on stored the code inside GitHub. At first I thought that someone has already solved this problem somehow, but doing an extensive search on the internet I found several angry GitHub users opening requests within the Support Community and even in the stack overflow. <img src="/static/assets/img/blog/gh-action-dropdown/comment-1.png" alt="comment-1" /> <img src="/static/assets/img/blog/gh-action-dropdown/comment-2.png" alt="comment-2" /> <img src="/static/assets/img/blog/gh-action-dropdown/comment-3.png" alt="comment-3" /> <img src="/static/assets/img/blog/gh-action-dropdown/comment-5.png" alt="comment-5" /> <img src="/static/assets/img/blog/gh-action-dropdown/comment-4.png" alt="comment-4" /> So I decided to create a solution for this, always thinking about simplicity and in a way that makes it easy to get this missing functionality. I started by creating an input array pattern using <code class="language-plaintext highlighter-rouge">commas</code> and using a <code class="language-plaintext highlighter-rouge">tag</code> (the selector) e.g <code class="language-plaintext highlighter-rouge">brackets</code> as the default value marker. Here is an example of what an input string would look like: <div class="language-yml highlighter-rouge"><div class="highlight"><pre class="highlight"><code>name: gh-action-dropdown-list-input on: workflow_dispatch: inputs: environment: description: 'Environment' required: true default: 'dev,staging,[uat],prod' </code></pre></div></div> Now the final question that would turn out to be the most complicated to deal with. How can I change the GitHub Actions interface to replace the input pattern we created earlier to a dropdown? The simplest answer I thought was to create a <code class="language-plaintext highlighter-rouge">chrome and firefox extension</code> that would do all this logic behind the scenes and replace the <code class="language-plaintext highlighter-rouge">HTML input element</code> with the <code class="language-plaintext highlighter-rouge">selected tag</code> containing the array values and leaving the tag value (selector) always as the default. All code was developed in pure JavaScript, open-source licensed under Apache 2.0 and available at https://github.com/arthurbdiniz/gh-action-dropdown-list-input. <h2 id="install-extension">Install extension</h2> <ul> <li><a href="https://chrome.google.com/webstore/detail/github-action-dropdown-in/deogklnblohhopmnkllaeinijefddcnm">Chrome</a></li> <li><a href="https://addons.mozilla.org/en-US/firefox/addon/github-action-dropdown-input/">Firefox</a></li> </ul> Once installed, the extension is ready to use and the final result we see is the Actions interface with drop-downs. :) <img src="/static/assets/img/blog/gh-action-dropdown/showcase-1.png" alt="showcase-1" /> <img src="/static/assets/img/blog/gh-action-dropdown/showcase-2.png" alt="showcase-2" /> <h3 id="configuring-selectors">Configuring selectors</h3> Go to the top right corner of the browser you are using and click on the extension logo. A screen will popup with tag options. Choose the right tags for you and save it. <blockquote> This action might require reloading the GitHub workflow tab. <img src="/static/assets/img/blog/gh-action-dropdown/config.png" alt="config" /> </blockquote> <hr /> Have fun using drop-downs inside GitHub. If you liked this project please share this post and if possible “star” within the repository. Also feel free to connect with me on LinkedIn: https://www.linkedin.com/in/arthurbdiniz <h1 id="references">References</h1> <ul> <li>https://github.community/t/can-workflow-dispatch-input-be-option-list/127338</li> <li>https://stackoverflow.com/questions/69296314/dropdown-for-github-workflows-input-parameters</li> </ul> Sat, 23 Oct 2021 00:00:00 +0000 https://www.arthurbdiniz.com/blog/gh-action-dropdown.html https://www.arthurbdiniz.com/blog/gh-action-dropdown.html GitHub Actions Workflows Upgrade Elasticsearch from 2.3 to 7.4 Sometimes it is common just to deploy a Elasticsearch domain at AWS and forget to keep it up-to-date with the latest version. After a while, you realize that it’s five major releases behind the stable version and there’s no straight path to upgrade. The idea of this post is to show the problems that you can face during this process of upgrading and also give you a more simplified way than following the AWS documentation. <h2 id="pre-requirements">Pre-requirements</h2> <ul> <li>ES domain v2.3</li> <li>ES domain v5.1 (Optional)</li> <li>IAM account with S3 and ES permissions</li> </ul> <hr /> <h2 id="from-23-to51">From 2.3 to 5.1</h2> According to the AWS documentation, the first upgrade needs to be done manually using the ES snapshot API. So to successfully upgrade, you will need to create an S3 bucket and a set of permissions that allow your ES domains to dump data into the bucket. To save your time, we have created a python script that communicates with the AWS API and setup everything for us. Focused on avoiding problems with your root domain, this script will create a brand new domain and execute all the steps in there. All the code can be found at: <a href="https://github.com/DNXLabs/es-auto-upgrade">es-auto-upgrade</a> <h4 id="script-steps">Script steps</h4> <ol> <li>Create an S3 bucket to store snapshot from 2.3 domain</li> <li>Create new Elasticsearch domain (Optional)</li> <li>Create IAMpermissions (Policy, Role, and attachments)</li> <li>Register snapshot in both domains</li> <li>Take snapshot from 2.3 and save it to S3 bucket</li> <li>Restore snapshot from S3 into the new domain</li> </ol> <h4 id="teardown">Teardown</h4> <ol> <li>Delete S3 bucket</li> <li>Delete IAMpermissions (Policy, Role, and detachments)</li> </ol> <h4 id="dependencies">Dependencies</h4> <ul> <li>Docker</li> </ul> <h4 id="variables">Variables</h4> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code># Elasticsearch export OLD_DOMAIN_NAME=test export AWS_REGION=ap-southeast-2 export NEW_DOMAIN_NAME=test-new export CREATE_NEW_DOMAIN=True # Optional export NEW_INSTANCE_TYPE=m5.xlarge.elasticsearch # Optional # S3 export BUCKET_NAME=es-automated-update # Optional </code></pre></div></div> To run all of these steps at once just run: <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code># Download the required variables, you should edit this file. wget https://raw.githubusercontent.com/DNXLabs/es-auto-upgrade/master/var.env docker run -it --env-file vars.env dnxsolutions/es-auto-upgrade upgrade.py </code></pre></div></div> When the script finishes running, you should have your brand new ES domain at version 5.1 with all your data from 2.3. <h2 id="from-51-to74">From 5.1 to 7.4</h2> From now on Amazon ES offers in-place Elasticsearch upgrades for domains that run versions 5.1 and later and we can proceed using the In-place along with the Elasticsearch reindex API to get to version 7.4. Currently, Amazon ES supports the following upgrade paths. <h4 id="51-to56">5.1 to 5.6</h4> <h4 id="56-to68">5.6 to 6.8</h4> <blockquote> Important: Indices created in version 6.x no longer support multiple mapping types. Indices created in version 5.x still support multiple mapping types when restored into a 6.x cluster. Check that your client code creates only a single mapping type per index. To minimize downtime during the upgrade from Elasticsearch 5.6 to 6.x, Amazon ES reindexes the .kibana index to .kibana-6, deletes .kibana, creates an alias named .kibana, and maps the new index to the new alias. </blockquote> <h4 id="68-to74">6.8 to 7.4</h4> <blockquote> Important: Elasticsearch 7.0 includes numerous breaking changes. Before initiating an in-place upgrade, we recommend taking a manual snapshot of the 6.8 domain, restoring it on a test 7.x domain, and using that test domain to identify potential upgrade issues. Like Elasticsearch 6.x, indices can only contain one mapping type, but that type must now be named _doc. As a result, certain APIs no longer require a mapping type in the request body (such as the _bulk API). For new indices, self-hosted Elasticsearch 7.x has a default shard count of one. Amazon ES 7.x domains retain the previous default of five. </blockquote> This next steps will repeat until you get to the latest ES version. <h4 id="1-reindex">1. Reindex</h4> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>docker run -it --env-file vars.env dnxsolutions/es-auto-upgrade reindex.py </code></pre></div></div> <h4 id="2-rolloutupgrade">2. Rollout Upgrade</h4> Go to your domain actions and select the Upgrade Domain button. <img src="/static/assets/img/blog/upgrade-es/rollout_1.png" alt="rollout_1" /> Then select the Upgrade checkbox and Submit. <img src="/static/assets/img/blog/upgrade-es/rollout_2.png" alt="rollout_2" /> AWS now will take care of the rest taking snapshots and upgrading to the next ES version. You can check the progress at the Upgrade History tab. <img src="/static/assets/img/blog/upgrade-es/rollout_2.png" alt="rollout_3" /> Once this step is finished, repeat the reindex and upgrade until you get to 7.4. <h2 id="troubleshooting">Troubleshooting</h2> Your domain sometimes might be ineligible for an upgrade or fail to upgrade for a wide variety of reasons. So AWS has provided one table showing the most common issues. https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-version-migration.html#upgrade-failures <h2 id="mapping-issues">Mapping Issues</h2> When you get to 7.4, you might notice some changes with the mapping. Since 2.3, a lot of new features were implemented, and this causes a lot of changes with mapping. So what could happen is that you have a variable <code class="language-plaintext highlighter-rouge">title</code> with called <code class="language-plaintext highlighter-rouge">id</code> and it has changed to <code class="language-plaintext highlighter-rouge">string</code>. What you can do it take the new mapping consulting the ES API at the endpoint. <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>https://$ES_DOMAIN_URL/<indice>/_mappin </code></pre></div></div> Then compare with the original mapping at version 2.3 and adapt to your needs. <div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code>file: mapping.json { "mappings":{ "properties":{ "style": { "type": "text", "analyzer": "keyword" } } } } </code></pre></div></div> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code># Apply the new mapping for index in <indice>; do curl -HContent-Type:application/json -XPUT "$ES_DOMAIN_URL/$index-new" -d @mapping.json done </code></pre></div></div> <hr /> <h4 id="string--text">String -> Text</h4> <div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code>{ "mappings":{ "properties":{ "id": { "type": "text" } } } } </code></pre></div></div> <h4 id="float--long">Float -> Long</h4> <div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code>{ "mappings":{ "properties":{ "price":{ "type":"long" } } } } </code></pre></div></div> <h4 id="completion-type">Completion type</h4> If you use completion type with <code class="language-plaintext highlighter-rouge">payload</code> and <code class="language-plaintext highlighter-rouge">output</code>, you may notice that since v5.1, ES does not support these two fields anymore. So to find a workaround for this, the following issue did this very well. <blockquote> Now, instead of returning the payload, the whole hit is returned from Elasticsearch, so you can use any regular property of the document. In order to save bandwidth and memory, it makes sense to return only properties which you care about, by using the “_source filtering” feature (see _source_include and _source_exclude features. </blockquote> https://github.com/elastic/elasticsearch-rails/issues/690#issuecomment-290706168 So, for example, this is one way of using context, <code class="language-plaintext highlighter-rouge">payload</code>, and <code class="language-plaintext highlighter-rouge">output</code>: <div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code>"brand_suggestion":{ "properties":{ "context":{ "properties":{ "visible_context":{ "type":"boolean" } } }, "input":{ "type":"completion", "analyzer":"simple", "preserve_separators":true, "preserve_position_increments":true, "max_input_length":50, "contexts":[ { "name":"visible_context", "type":"CATEGORY", "path":"is_visible" } ] }, "output":{ "type":"text", "fields":{ "keyword":{ "type":"keyword", "ignore_above":256 } } }, "payload":{ "properties":{ "id":{ "type":"long" }, "urlKey":{ "type":"text", "fields":{ "keyword":{ "type":"keyword", "ignore_above":256 } } } } } } } </code></pre></div></div> <h2 id="conclusion">Conclusion</h2> We hope we have helped with some insights into your Amazon Elasticsearch Service migration. It is not an easy job, and if you have a big distributed cluster, you may want to use other tools to handle this amount of workload. <h2 id="references">References</h2> <ul> <li>https://www.intercom.com/blog/upgrading-elasticsearch</li> <li>https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-reindex.html</li> <li>https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-version-migration.html</li> <li>https://aws.amazon.com/blogs/database/in-place-version-upgrades-for-amazon-elasticsearch-service/</li> </ul> Wed, 24 Jun 2020 00:00:00 +0000 https://www.arthurbdiniz.com/blog/upgrade-es-2.3-to-7.4.html https://www.arthurbdiniz.com/blog/upgrade-es-2.3-to-7.4.html AWS Elasticsearch Upgrade GSoC Community Bonding Experience Following the approval of <a href="https://summerofcode.withgoogle.com/">Google Summer of Code</a> as the student selected on the development of the <a href="">Cloud Image Finder</a>, a web application that will make it easier for users to find official Debian cloud images, with <code class="language-plaintext highlighter-rouge">Lucas Kanashiro</code> and <code class="language-plaintext highlighter-rouge">Bastian Blank</code> as mentors. A <a href="https://summerofcode.withgoogle.com/how-it-works/#timeline">community bounding</a> period begins where students from each organization spend a period from <code class="language-plaintext highlighter-rouge">May 6 to May 27, 2019</code>, knowing more about their organization. That way I would like to document my experience with Debian during this time. In the first week talking to my mentor <code class="language-plaintext highlighter-rouge">Lucas Kanashiro</code>, i started my communication with the IRC channels where I saw that the community is very active and collaborative, I had some problems to keep myself active as IRC because of the firewall of my university and I ended up going to a cloud solution of IRC, where at the moment i am using the free client <a href="https://beta.alwyzon.com">alwyzon</a>. About my channel communication, as soon as I joined the #debian-cloud channel i was very happy and felt embraced by the community inside the project, seeing people willing to help in what is necessary and always giving me feedbacks. Another way of communication that I found to be very important are <a href="https://lists.debian.org/completeindex.html">mailing lists</a> in which all content is stored in Debian`s infrastructure and has greater visibility. That way I learned two things, if you just want to take a quick question or talk and that are very specific problems, IRC channels will help you, in the case of notify the community and leave it saved you should go to mailing list. At the end of community bonding I was already to send and reply to emails and using IRC to communicate and in order to elicitate project requirements we use the <code class="language-plaintext highlighter-rouge">prototyping technique</code>. We started from a <a href="https://salsa.debian.org/cloud-team/image-finder/wikis/Low-Fidelity-Prototype">Low Fidelity Prototype</a>, which was used to validate basic concepts and also gather some feedback from the Cloud Team. After an evaluation we came up with a <a href="https://salsa.debian.org/cloud-team/image-finder/wikis/High-Fidelity-Prototype">High Fidelity Prototype</a>, where we applied the required modifications raised in the last step. All of this has been documented in <a href="https://salsa.debian.org/cloud-team/image-finder/issues">issues</a> within our <a href="https://salsa.debian.org/cloud-team/image-finder">repository</a> and <a href="https://salsa.debian.org/cloud-team/image-finder/wikis">wiki</a>. Finally last week I had a meeting with my mentor Lucas Kanashiro where we discussed about the features prioritization based on cloud team feedback over low-fidelity prototype, where this meeting resulted in making some decisions before starting the coding phase. We noticed the necessity to perform some steps prior starting the code phase. The first one is to design the database schema based on the output artifacts in the <a href="https://salsa.debian.org/cloud-team/debian-cloud-images/pipelines">build</a> of the cloud images we would model our database. Cloud Image Finder Next Steps: <ul> <li><a href="https://salsa.debian.org/cloud-team/image-finder/wikis/Database-Schema">Database Schema</a></li> <li>Architecture Diagram</li> <li>Start Code</li> </ul> Mon, 27 May 2019 00:00:00 +0000 https://www.arthurbdiniz.com/blog/gsoc-community-bonding.html https://www.arthurbdiniz.com/blog/gsoc-community-bonding.html GSoC Communiy Welcome Welcome, in this blog I intend to present my contributions to Debian Project and update about the Cloud Image Finder project from Google Summer of Code 2019. Thu, 23 May 2019 00:00:00 +0000 https://www.arthurbdiniz.com/blog/welcome.html https://www.arthurbdiniz.com/blog/welcome.html Welcome Introduction

Original Source | Taken Source