Refactoring and minor improvements

README.md

@@ -16,16 +16,16 @@
 
 
 ## Installation
-- Download or [build](#build-debian-packages) the Debian packages
-- Install the `homcc` client via: ```sudo apt install ./target/homcc.deb```
-- Install the `homccd` server via: ```sudo apt install ./target/homccd.deb```
+- [Download](https://github.com/celonis/homcc/releases) the latest release or [build](#build-debian-packages) the Debian packages yourself
+- Install the `homcc` client via: ```sudo apt install ./homcc.deb```
+- Install the `homccd` server via: ```sudo apt install ./homccd.deb```
 
-**Note**: Currently, installing both packages leads to an issue with conflicting files. Therefore, to install the second package, use `sudo dpkg -i --force-overwrite ./target/{package.deb}`!
+**Note**: Currently, installing both packages leads to an issue with conflicting files. Therefore, to install the second package, use `sudo dpkg -i --force-overwrite ./{package.deb}`!
 
 
 ## Documentation
 - TODO:
-  - Brief overview of what `homcc` is and why it exists: distribtued compilation -> faster build times, modern alternative
+  - Brief overview of what `homcc` is and why it exists: distributed compilation -> faster build times, modern alternative
   - Differences to `distcc`: thin connection as priority, caching, local pre-processing
   - Description of client-server interaction
   - Description of server-side caching
@@ -34,7 +34,7 @@
 ## Usage and Configuration
 
 ### Client: `homcc`
-- Follow the client [Installation](#Installation) guide
+- Follow the client [Installation](#installation) guide
 - Find usage description and client defaults: `homcc --help`
 - Overwrite defaults via a `client.conf` configuration file:
   - Possible `client.conf` locations:
@@ -43,19 +43,21 @@
     - `~/.config/homcc/client.conf`
     - `/etc/homcc/client.conf`
   - Possible `homcc` configurations:
-    - `verbose`: enable a verbose mode by specifying `True` which implies detailed and colored logging of debug messages
     - `compiler`: compiler if none is explicitly specified via the CLI
     - `timeout`: timeout value in seconds for each remote compilation attempt
     - `compression`: compression algorithm, choose from `{lzo, lzma}`
     - `profile`: `schroot` environment profile that will be used on the server side for compilations
+    - `log_level`: detail level for log messages, choose from `{DEBUG,INFO,WARNING,ERROR,CRITICAL}`
+    - `verbose`: enable a verbose mode by specifying `True` which implies detailed and colored logging of debug messages, can be combined with `log_level`
   - Example:
     ```
     # homcc: example client.conf
-    verbose=True
     compiler=g++
     timeout=180
     compression=lzo
     profile=schroot_environment
+    log_level=DEBUG
+    verbose=True
     ```
 - Specify your remote compilation server in a `hosts` file or in the `$HOMCC_HOSTS` environment variable:
   - Possible `hosts` file locations:
@@ -68,26 +70,27 @@
       - `HOST`: TCP connection to specified `HOST` with default port `3633`
       - `HOST:PORT`: TCP connection to specified `HOST` with specified `PORT`
     - `HOST/LIMIT` format:
-      - define any of the above `HOST` format with an additional `LIMIT` parameter that specifies the maximum connection limit to the corresponding `HOST`
-      - it is advised to always specify your `LIMIT`s as they will otherwise default to 2 and only enable minor levels of concurrency
+      - Define any of the above `HOST` format with an additional `LIMIT` parameter that specifies the maximum connection limit to the corresponding `HOST`
+      - It is advised to always specify your `LIMIT`s as they will otherwise default to 2 and only enable minor levels of concurrency
     - `HOST,COMPRESSION` format:
-      - define any of the above `HOST` or `HOST/LIMIT` format with an additional `COMPRESSION` algorithm information
-      - choose from:
+      - Define any of the above `HOST` or `HOST/LIMIT` format with an additional `COMPRESSION` algorithm information
+      - Choose from:
         - `lzo`: Lempel-Ziv-Oberhumer compression algorithm
         - `lzma`: Lempel-Ziv-Markov chain algorithm
-      - per default, no compression is used as it is usually not necessary for high bandwidth connections
+      - No compression is used per default, specifying `lzo` is however advised
+  - **WARNING**: Currently do not include `localhost` in your hosts file!
   - Example:
     ```
     # homcc: example hosts
     localhost/12
     127.0.0.1:3633/21
     [::1]:3633/42,lzo
     ```
-- Use `homcc` in your `conan` profile by specifying: `CCACHE_PREFIX=homcc`
+- Use `homcc` by specifying `CCACHE_PREFIX=homcc` in your `conan` profile and only have `CONAN_CPU_COUNT` smaller or equal to the sum of all remote host limits, e.g. `≤ 12+21+42` for the example above!
 
 
 ### Server: `homccd` 
-- Follow the server [Installation](#Installation) guide
+- Follow the server [Installation](#installation) guide
 - Find usage description and server defaults: `homccd --help`
 - Overwrite defaults via a `server.conf` configuration file:
   - Possible `server.conf` locations:
@@ -100,14 +103,14 @@
     - `port`: TCP port to listen on
     - `address`: IP address to listen on
     - `log_level`: detail level for log messages, choose from `{DEBUG,INFO,WARNING,ERROR,CRITICAL}`
-    - `verbose`: enable a verbose mode by specifying `True` which implies detailed and colored logging of debug messages
+    - `verbose`: enable a verbose mode by specifying `True` which implies detailed and colored logging of debug messages, can be combined with `log_level`
   - Example:
     ```
     # homccd: example server.conf
     limit=64
-    log_level=DEBUG
     port=3633
     address=0.0.0.0
+    log_level=DEBUG
     verbose=True
     ```
 - \[Optional]: Setup your `chroot` environments at `/etc/schroot/schroot.conf` or in the<br/>
@@ -146,14 +149,22 @@
 - Format a specified python file: `black ./path/to/file.py`
 
 ### Build Debian packages
-- Run `make all` in the repository root to build the `server` and `client` target
+- Install required tools:<br/>
+  ```
+  sudo apt install -y \
+    python3 python3-dev python3-pip python3-venv python3-all \
+    dh-python debhelper devscripts dput software-properties-common \
+    python3-distutils python3-setuptools python3-wheel python3-stdeb \
+    liblzo2-dev
+  ```
+- Run `make homcc`, `make homccd` or `make all` to build the corresponding `client` and `server` package
 - The generated `.deb` files are then contained in the `./target/` directory
 
 
 ### `schroot` testing setup for Debian systems
-- Install necessary tools: `sudo apt install schroot debootstrap`
+- Install required tools: `sudo apt install schroot debootstrap`
 - Create `chroot` environment:
-  - Download and install selected distribution at your desired location, e.g. `Ubuntu 22.04 Jammy Jellyfish` from [Ubuntu Releases](https://wiki.ubuntu.com/Releases) at `/var/chroot/`:<br/>
+  - Download and install selected distribution to your desired location, e.g. `Ubuntu 22.04 Jammy Jellyfish` from [Ubuntu Releases](https://wiki.ubuntu.com/Releases) at `/var/chroot/`:<br/>
     `sudo debootstrap jammy /var/chroot/jammy http://archive.ubuntu.com/ubuntu`
   - Configure the environment by creating a corresponding file in the `/etc/schroot/chroot.d/` directory or by appending it to `/etc/schroot/schroot.conf`, e.g. by replacing `USERNAME` in `jammy.conf`:<br/>
     ```

homcc/client/client.py

@@ -14,8 +14,6 @@
 from homcc.client.parsing import ConnectionType, Host, parse_host
 from homcc.common.arguments import Arguments
 from homcc.common.messages import ArgumentMessage, DependencyReplyMessage, Message
-from homcc.common.compression import Compression, NoCompression
-
 
 logger = logging.getLogger(__name__)
 
@@ -78,45 +76,45 @@ class TCPClient:
     """Wrapper class to exchange homcc protocol messages via TCP"""
 
     DEFAULT_PORT: int = 3633
-    DEFAULT_TIMEOUT: float = 180
+    DEFAULT_BUFFER_SIZE_LIMIT: int = 65_536  # default buffer size limit of StreamReader is 64 KiB
     DEFAULT_OPEN_CONNECTION_TIMEOUT: float = 5
 
-    def __init__(self, host: Host, buffer_limit: Optional[int] = None):
+    def __init__(self, host: Host):
         connection_type: ConnectionType = host.type
 
         if connection_type != ConnectionType.TCP:
             raise ValueError(f"TCPClient cannot be initialized with {connection_type}!")
 
         self.host: str = host.host
         self.port: int = host.port or self.DEFAULT_PORT
-
-        # default buffer size limit of StreamReader is 64 KiB
-        self.buffer_limit: int = buffer_limit or 65_536
-
-        self.compression = Compression.from_name(host.compression) if host.compression is not None else NoCompression()
+        self.compression = host.compression
 
         self._data: bytes = bytes()
         self._reader: asyncio.StreamReader
         self._writer: asyncio.StreamWriter
 
     async def __aenter__(self) -> TCPClient:
         """connect to specified server at host:port"""
-        logger.debug("Connecting to %s:%i", self.host, self.port)
-        self._reader, self._writer = await asyncio.wait_for(
-            asyncio.open_connection(host=self.host, port=self.port, limit=self.buffer_limit),
-            timeout=self.DEFAULT_OPEN_CONNECTION_TIMEOUT,
-        )
+        logger.debug("Connecting to '%s:%i'.", self.host, self.port)
+        try:
+            self._reader, self._writer = await asyncio.wait_for(
+                asyncio.open_connection(host=self.host, port=self.port, limit=self.DEFAULT_BUFFER_SIZE_LIMIT),
+                timeout=self.DEFAULT_OPEN_CONNECTION_TIMEOUT,
+            )
+        except asyncio.TimeoutError as error:
+            logger.warning("Connection establishment to '%s:%s' timed out.", self.host, self.port)
+            raise error from None
         return self
 
     async def __aexit__(self, *_):
         """disconnect from server and close client socket"""
-        logger.debug("Disconnecting from %s:%i", self.host, self.port)
+        logger.debug("Disconnecting from '%s:%i'.", self.host, self.port)
         self._writer.close()
         await self._writer.wait_closed()
 
     async def _send(self, message: Message):
         """send a message to homcc server"""
-        logger.debug("Sending %s to %s:%i:\n%s", message.message_type, self.host, self.port, message.get_json_str())
+        logger.debug("Sending %s to '%s:%i':\n%s", message.message_type, self.host, self.port, message.get_json_str())
         self._writer.write(message.to_bytes())  # type: ignore[union-attr]
         await self._writer.drain()  # type: ignore[union-attr]
 
@@ -134,30 +132,30 @@ async def send_dependency_reply_message(self, dependency: str):
     async def receive(self) -> Message:
         """receive data from homcc server and convert it to Message"""
         #  read stream into internal buffer
-        self._data += await self._reader.read(self.buffer_limit)
+        self._data += await self._reader.read(self.DEFAULT_BUFFER_SIZE_LIMIT)
         bytes_needed, parsed_message = Message.from_bytes(bytearray(self._data))
 
         # if message is incomplete, continue reading from stream until no more bytes are missing
         while bytes_needed > 0:
-            logger.debug("Message is incomplete by %i bytes", bytes_needed)
+            logger.debug("Message is incomplete by #%i bytes.", bytes_needed)
             self._data += await self._reader.read(bytes_needed)
             bytes_needed, parsed_message = Message.from_bytes(bytearray(self._data))
 
         # manage internal buffer consistency
         if bytes_needed == 0:
             # reset the internal buffer
-            logger.debug("Resetting internal buffer")
+            logger.debug("Resetting internal buffer.")
             self._data = bytes()
         elif bytes_needed < 0:
             # remove the already parsed message
-            logger.debug("Additional data of %i bytes in buffer", abs(bytes_needed))
+            logger.debug("Additional data of #%i bytes in buffer.", abs(bytes_needed))
             self._data = self._data[len(self._data) - abs(bytes_needed) :]
 
         if not parsed_message:
             raise ClientParsingError("Received data could not be parsed to a message!")
 
         logger.debug(
-            "Received %s message from %s:%i:\n%s",
+            "Received %s message from '%s:%i':\n%s",
             parsed_message.message_type,
             self.host,
             self.port,

homcc/client/compilation.py

@@ -15,7 +15,7 @@
     TCPClient,
 )
 from homcc.client.errors import HostsExhaustedError, RemoteCompilationError, UnexpectedMessageTypeError
-from homcc.client.parsing import ConnectionType, ClientConfig, Host
+from homcc.client.parsing import ClientConfig, Host
 from homcc.common.arguments import Arguments, ArgumentsExecutionResult
 from homcc.common.hashing import hash_file_with_path
 from homcc.common.messages import (
@@ -28,27 +28,34 @@
 
 logger = logging.getLogger(__name__)
 
+DEFAULT_COMPILATION_REQUEST_TIMEOUT: float = 180
+
 
 async def compile_remotely(arguments: Arguments, hosts: List[str], config: ClientConfig) -> int:
     """main function to control remote compilation"""
 
-    # try to connect to 3 different remote compilation hosts before giving up
+    # try to connect to 3 different hosts before falling back to local compilation
     for host in HostSelector(hosts, 3):
-        timeout: float = config.timeout or 180
-        profile: Optional[str] = config.profile
-        host.compression = host.compression or config.compression
-
-        if host.type == ConnectionType.LOCAL:
+        # execute compilation requests for localhost directly
+        if host.type.is_local():
             logger.info("Compiling locally:\n%s", arguments)
             return compile_locally(arguments)
 
+        timeout: float = config.timeout or DEFAULT_COMPILATION_REQUEST_TIMEOUT
+        profile: Optional[str] = config.profile
+
+        # overwrite host compression if none was specified
+        host.compression = host.compression or config.compression
+
         try:
             return await asyncio.wait_for(compile_remotely_at(arguments, host, profile), timeout=timeout)
 
-        except (asyncio.TimeoutError, ConnectionError) as error:
+        except ConnectionError as error:
             logger.warning("%s", error)
+        except asyncio.TimeoutError:
+            logger.warning("Compilation request for host '%s' timed out.", host.host)
 
-    raise HostsExhaustedError(f"All hosts {hosts} are exhausted!")
+    raise HostsExhaustedError(f"All hosts {hosts} are exhausted.")
 
 
 async def compile_remotely_at(arguments: Arguments, host: Host, profile: Optional[str]) -> int:
@@ -59,7 +66,7 @@ async def compile_remotely_at(arguments: Arguments, host: Host, profile: Optiona
     async with TCPClient(host) as client:
         await client.send_argument_message(remote_arguments, os.getcwd(), dependency_dict, profile)
 
-        # invert dependency dictionary
+        # invert dependency dictionary to access dependencies via hash
         dependency_dict = {file_hash: dependency for dependency, file_hash in dependency_dict.items()}
 
         host_response: Message = await client.receive()

homcc/client/main.py

@@ -19,6 +19,7 @@
 from homcc.client.errors import RecoverableClientError, RemoteCompilationError  # pylint: disable=wrong-import-position
 from homcc.client.parsing import (  # pylint: disable=wrong-import-position
     ClientConfig,
+    LogLevel,
     load_config_file,
     load_hosts,
     parse_cli_args,
@@ -38,26 +39,37 @@
 def main():
     # load and parse arguments and configuration information
     homcc_args_dict, compiler_arguments = parse_cli_args(sys.argv[1:])
-    client_config: ClientConfig = parse_config(load_config_file())
+    homcc_config: ClientConfig = parse_config(load_config_file())
     logging_config: LoggingConfig = LoggingConfig(
         config=FormatterConfig.COLORED,
         formatter=Formatter.CLIENT,
         destination=FormatterDestination.STREAM,
     )
 
-    # VERBOSE; enables verbose mode
-    if homcc_args_dict["verbose"] or client_config.verbose:
+    # LOG_LEVEL and VERBOSITY
+    log_level: str = homcc_args_dict["log_level"]
+
+    # verbosity implies debug mode
+    if (
+        homcc_args_dict["verbose"]
+        or homcc_config.verbose
+        or log_level == "DEBUG"
+        or homcc_config.log_level == LogLevel.DEBUG
+    ):
         logging_config.config |= FormatterConfig.DETAILED
         logging_config.level = logging.DEBUG
 
+    # overwrite verbose debug logging level
+    if log_level is not None:
+        logging_config.level = LogLevel[log_level].value
+    elif homcc_config.log_level is not None:
+        logging_config.level = int(homcc_config.log_level)
+
     setup_logging(logging_config)
 
     # COMPILER; default: "cc"
-    compiler: Optional[str] = compiler_arguments.compiler
-
-    if not compiler:
-        compiler = client_config.compiler
-        compiler_arguments.compiler = compiler
+    if compiler_arguments.compiler is None:
+        compiler_arguments.compiler = homcc_config.compiler
 
     # SCAN-INCLUDES; and exit
     if homcc_args_dict["scan_includes"]:
@@ -74,15 +86,15 @@ def main():
     profile: Optional[str] = homcc_args_dict["profile"]
 
     if homcc_args_dict["no_profile"]:
-        client_config.profile = None
+        homcc_config.profile = None
     elif profile:
-        client_config.profile = profile
+        homcc_config.profile = profile
 
     # TIMEOUT
     timeout: Optional[float] = homcc_args_dict["timeout"]
 
     if timeout:
-        client_config.timeout = timeout
+        homcc_config.timeout = timeout
 
     if compiler_arguments.is_linking_only():
         logger.debug("Linking [%s] to %s", ", ".join(compiler_arguments.object_files), compiler_arguments.output)
@@ -91,7 +103,7 @@ def main():
     # try to compile remotely
     if compiler_arguments.is_sendable():
         try:
-            sys.exit(asyncio.run(compile_remotely(compiler_arguments, hosts, client_config)))
+            sys.exit(asyncio.run(compile_remotely(compiler_arguments, hosts, homcc_config)))
 
         # exit on unrecoverable errors
         except RemoteCompilationError as error: