Add custom executable (output object) format

stevenwdv · 2023-02-17T21:44:28.980Z

Hi, I’m creating a backend for my own custom virtual processor, and now that the plain text assembly output with llc works, I’d like to get clang to output my custom executable format. Thus my question: How would I go about doing this? Do I need a custom MCObjectWriter, MCObjectStreamer, etc., or changes to lld? Can I reuse the existing linker instead of writing my own? Ideally, I’d like to use the custom object format only for the final executable, not for intermediate object files, I don’t care about the format of these, as long as I can compile a program containing multiple source code files and produce the custom executable in the end. I don’t need support for dynamic libraries (or static libraries, for that matter).

I really hope someone can give me some hints, thanks!

pogo59 · 2023-02-21T16:21:11.737Z

LLVM supports several different object formats: ELF, COFF, MachO, and probably some variants. If you are happy with one of those formats for your relocatable object files, then the only compiler changes would be to have your target hook up the correct object file format.

If you want a custom executable format, I can see two options. One, try to leverage an existing linker to emit your custom format. Two, use an existing linker to output a standard format, then write a conversion tool. This means an extra post-link processing step, but might be simpler in the short term.

tschuett · 2023-02-21T16:35:08.155Z

The newest member to the family is XCOFF.

HLSL circumvents the binary backend:

[RFC] Adding HLSL and DirectX support to Clang & LLVM Clang Frontend

Hello all, The HLSL compiler community is interested in contributing HLSL, DirectX and Vulkan graphics support to Clang and LLVM. Why do we want to do this? The existing HLSL compiler, the DirectX Shader Compiler (DXC), is a fork of LLVM/Clang 3.7 and is developed open source on GitHub (https://github.com/microsoft/DirectXShaderCompiler) by Microsoft and a diverse community of open source partners. We plan to update our compiler to the latest LLVM. This plan is motivated by our efforts to bri…

stevenwdv · 2023-02-21T22:45:06.867Z

Thanks all!

pogo59:

If you want a custom executable format, I can see two options. One, try to leverage an existing linker to emit your custom format. […]

So this first method would need changes in ./lld/, I guess? In this case, will (could) the intermediate object files still be ELF or something? (Or is there some more neutral format?) Not sure if LLD has some intermediate representation after it links the object files, such that this could be easily transformed into any executable.

tschuett:

The newest member to the family is XCOFF.

Does LLD also support this? I can’t find it in ./lld/.

tschuett:

HLSL circumvents the binary backend:

I see, because it wants to output something like LLVM IR I guess. In this case, would it be impossible to compile a project with multiple source (e.g. .c) files, because the linker is skipped, if I understand correctly? Also, I’d like to have absolute memory addresses (e.g. for globals) in the final executable, so I guess I’d need the linker for that, right?

pogo59 · 2023-02-22T03:23:20.289Z

stevenwdv:

So this first method would need changes in ./lld/, I guess? In this case, will (could) the intermediate object files still be ELF or something?

If you want a custom object-file format, you will need to modify something. I’m not really familiar with the internals of LLD, so I couldn’t say whether it would be better to have the compiler emit a new format and add a whole new “flavor” to LLD, or have the compiler emit an existing format and persuade LLD to turn that into a different output format. But if you want the final executable to be in a custom format, you will need to do one of those things.

tschuett · 2023-02-22T06:37:06.217Z

IBM uses XCOFF for AIX. There are existing toolchains. They are just starting to add AIX support to LLVM.

stevenwdv · 2023-05-30T11:06:47.492Z

In the end, it turns out that adding support for a new binary format requires a lot of code changes. Notably, lld has almost no shared code between different binary formats so adding a new one requires re-implementing everything, including symbol resolving and relocations.

Hence, I opted to just use ELF (adding support for the relocations for my architecture), and then create a separate tool to convert that to the output I want (currently I integrated it with llvm-objcopy but I think I might separate it). This way, I can just use the existing ELF lld code for creating the object files and resolving symbols and (static) relocations.

This required doing the following (XXX = your target):

Set your default binary format to ELF in getDefaultFormat in llvm/lib/TargetParser/Triple.cpp
llvm/include/llvm/BinaryFormat:
- Add your architecture (EM_XXX) to the Machine architectures enum in ELF.h
- Add your relocation types to ELFRelocs/XXX.def (for my simple architecture I only needed R_XXX_NONE and R_XXX_32)
- Add the relocation types to ELF.h using enum { #include "ELFRelocs/XXX.def" };
Add the relocation types to llvm::object::getELFRelocationTypeName in llvm/lib/Object/ELF.cpp
llvm/lib/Targets/XXX/MCTargetDesc:
- Add XXXELFObjectWriter.cpp, inheriting from MCELFObjectTargetWriter
  - This needs to implement getRelocType. For me it seems to be enough to just translate the generic FK_Data_4 into my ELF::R_XXX_32 relocation, as I only have one constant size and no PC-relative relocations
- Make XXAsmBackend::createObjectTargetWriter return your XXXELFObjectWriter
  - Also, you need to implement fixupNeedsRelaxation and applyFixup. For me, fixupNeedsRelaxation just returns false and applyFixup shouldn’t need to do anything (Value is always 0), because I set HasRelocationAddend to true in the constructor of my XXXELFObjectWriter, so the addend doesn’t need to be written inline in the code, which is what applyFixup would do
In LLVMInitializeXXXTargetMC, call TargetRegistry::RegisterELFStreamer with a function that calls createELFStreamer
lld/ELF:
- Add Arch/XXX.cpp, where getRelExpr for me just returns R_ABS for my R_XXX_32, because I do not support PC-relative addressing, and relocate applies the relocation, for me checkUInt + write32le. You may want to support getImplicitAddend as well (e.g. read32be) to read an implicit written by XXXAsmBackend::applyFixup
- Add a case for your target in elf::getTarget in Target.cpp
And then we have the conversion tool, which I currently implemented in llvm-objcopy

Regarding relocations, the generic fixup kind (e.g. FK_Data_4) usually originates from your XXXMCCodeEmitter, which can add fixups for MCExprs during encodeInstruction. These MCExprs originate from XXXAsmPrinter::emitInstruction.

fzakaria · 2023-06-28T17:52:12.125Z

Thank you so much for this. If you have a Phabricator commit or something on github to cross-reference your response here that would be appreciated.

I am also exploring a new object file format (Exploring alternative object file formats)

I will likely look into it and see what changes are needed.
If you have any tips, knowledge or notes on the subject I’d be happy to have them

fzakaria · 2023-07-19T18:16:50.080Z

I started a branch pursuing this – if you are interested in following along DM me.

stevenwdv · 2023-08-07T12:04:24.418Z

If you have a Phabricator commit or something on github to cross-reference your response here that would be appreciated.

I will eventually update my code to GitHub, but I don’t think it’s fully ready yet.

Stoorx · 2023-08-07T18:55:49.912Z

Currently I am working on refactoring in these modules to do EXACTLY the same task. The aim is to make implementation of new targets easier and maintainable in general.