Installation and configuration of Red Hat Satellite 6: tips, workarounds, fixes and links

This post is a work in progress. I will keep track of the issues that I ran into while installing and configuring Red Hat Satellite 6 on Enterprise Linux 7.

Satellite as single instance

Are you running Satellite as a single instance to do "everything"? Then enable its DHCP, DNS and TFTP capsules (smart proxies)!

Repo issue on content host

In case of repo issues on a content host, run yum clean all and yum update first, before debugging further.

New content view created

When you create a Content View, be patient and let your repositories sync first.

Red Hat Common repository

Remember to add the Red Hat Common repository to the Content View of your VMs.

Satellite 6.0 specific

Satellite + VMware + "Unlimited Guest Subscription"

How to register RHEV / VMWare guest systems to Red Hat Satellite v6 to use 'Red Hat Unlimited Guest Subscription' subscriptions? by Mario Gamboa

Missing repos on client

Created a Content View but the client isn't showing all the repositories that you added? On the client:

  1. list all available subscriptions and look for the missing one: subscription-manager list --available --all
  2. add the missing subscription using its Pool ID: subscription-manager attach --pool=1234567890
  3. verify the subscription has been added: subscription-manager list --available --consumed
  4. see issue #2

If the repositories for your Content View aren't added/enabled automatically with subscription-manager register --org="Your_Organization" --activationkey="Your_Activation_Key" and subscription-manager attach --auto, in the Satellite 6 web interface go to Content → Activation Keys. Select your activation key and list the subscriptions. Make sure the subscriptions for your non-Red Hat repositories are added to the activation key.

RHEL 5 machines installed with Satellite 6 can't "consume" custom repositories

yum fails with [Errno -3] Error performing checksum: this is caused by the SHA-256 checksums used by Pulp for the repository. yum shipped with RHEL 5 doesn't support SHA256 yet. You can get SHA256 support by installing python-hashlib. Unfortunately, this package is not included in the Kickstart repository so you can't simply include it in the @packages list of your Kickstart file.

  1. Disable all your custom repositories
  2. Enable the Red Hat repositories
  3. Install python-hashlib
  4. Enable the custom repositories again
<%
  rhel_compatible = @host.operatingsystem.family == 'Redhat' && @host.operatingsystem.name != 'Fedora'
  os_major = @host.operatingsystem.major.to_i
  section_end = (rhel_compatible && os_major <= 5) ? '' : '%end'
%>

# Workaround for RHEL5 yum's SHA-256 incompatibility
# see https://bugzilla.redhat.com/show_bug.cgi?id=1009578 and
# https://bugzilla.redhat.com/show_bug.cgi?id=1127402
echo "Workaround for SHA256 incompatibility in RHEL5 yum"
subscription-manager repos --disable="*Default_Organization_*"
subscription-manager repos --enable="rhel-<%= @host.operatingsystem.major %>-*"
yum -y install python-hashlib
subscription-manager repos --enable="*Default_Organization_*"

Trying to publish new version of a Content View results in error Duplicate resource: Default_Organization-Example

This is a known bug and is supposed to be solved in Satellite 6.1, the fix won't be backported to 6.0. To fix the issue in Satellite 6.0, follow the steps from Mike McCune's post.

Failed to generate PXELinux template: private methodtemplate' called for nil:NilClass` when trying to cancel a build

This cryptic error is caused by a missing PXE template.

  1. In the Satellite 6 web interface go to Administer → Locations → Your Location → Templates and make sure that every template starting with "PXE" is added to your Location.
  2. Repeat the steps of #1 for Administer → Organization → Your Organization → Templates
  3. According to Foreman bug report #2215 "PXE Localboot Default" is enough, but it didn't fix my issue. When I enabled debugging in Foreman, I came by the query SELECT "config_templates".* FROM "config_templates" WHERE "config_templates"."id" IN (49, 46, 44, 14, 15, 1, 39, 47, 45, 41, 40, 35, 33, 34, 28, 36, 30, 37, 48) AND "config_templates"."name" = 'PXELinux default local boot' ORDER BY config_templates.name LIMIT 1. So the missing template that caused days of headache was actually called PXELinux default local boot

Can't start the celerybeat service

systemctl start pulp_celerybeat leads to:

celery: ERROR: Pidfile (celerybeat.pid) already exists.
celery: Seems we're already running? (pid: 1476)
celery: celery beat v3.1.11 (Cipater) is starting.
systemd: pulp_celerybeat.service: main process exited, code=exited, status=73/n/a
systemd: Unit pulp_celerybeat.service entered failed state.

.

The pid file is stored in /var/lib/pulp/celery/celerybeat.pid. Remove it and run systemctl start pulp_celerybeat.

Can't create a new host because the PTR record already exists

  1. Find the offending record in /var/named/dynamic/db.100.168.192.in-addr.arpa and
  2. delete the PTR record with nsupdate:

    echo "server 127.0.0.1 update delete 100.100.168.192.in-addr.arpa PTR host.example.com send " | nsupdate -k /etc/rndc.key

Can't execute any of the rpm subcommands with pulp-admin

Install the missing packages packages: yum install pulp-admin-client pulp-rpm-admin-extensions.noarch pulp-rpm-handlers.noarch as described in How to deploy and use pulp-admin.

Warning: The pulp-admin command can easily corrupt your database or put the pulp or katello databases out of sync. The use of pulp-admin outside the scope and direction of Red Hat Global Support Services is unsupported.

Remove all tasks in error state with foreman-rake console

ForemanTasks::Task.where(:result => :error).destroy_all. Change :result => :error to :state => :pending to remove pending tasks. See Red Hat KB solution #1381053.

Trying to change a host's Content View via https://satellite.example.com/hosts/hostname.example.com/edit fails with the message Katello::Resources::Candlepin::Consumer: 410 Gone {"displayMessage":"Unit xxx has been deleted","requestUuid":"yyy","deletedId":"xxx"} (GET /candlepin/consumers/xxx)

Remove the host from the Foreman Postgres database:

  1. su - postgres
  2. psql foreman
  3. foreman=# select id, name from katello_systems where name = 'host.example.com', note down the ID of the host
  4. foreman=# delete from katello_system_activation_keys where system_id = 123, 123 is the ID we got in the previous query
  5. foreman=# delete from katello_system_errata where system_id = 123;
  6. foreman=# delete from katello_systems where name = 'host.example.com'

Now you can update the Content View of the host.

Similar as above: to remove all hosts that use a Content View that's causing 410: Gone errors

  1. su - postgres
  2. psql foreman
  3. foreman=# select id, name from katello_content_views where name like '%content view%';, note down the ID of the host
  4. foreman=# delete from katello_system_activation_keys where (system_id) in (select id from katello_systems where content_view_id = 123);, 123 is the ID we got in the previous query
  5. foreman=# delete from katello_system_errata where (system_id) in (select id from katello_systems where content_view_id = 123);
  6. foreman=# delete from katello_system_repositories where (system_id) in (select id from katello_systems where content_view_id = 123);
  7. foreman=# delete from katello_systems where content_view = 123;

Search queries for Content Hosts view

content_view:"Your content view name" AND (environment:"your environment")

Satellite clients can't install packages because the URL in /etc/yum.repos.d/*.repo is incorrect

(Re-)run subscription-manager config --rhsm.baseurl=https://satellite.example.com/pulp/repos to correct the URL.

Tagged , , , , , , ,

Fixed NIC names (eth\*) on RHEL and CentOS

Tristan Terpelle Summary: How to go back to the classic naming of eth devices on EL7

EL7 uses systemd magic to give network devices consistent and predictable names based on firmware, topology, and location information. But what if you just want the NICs in your VMware virtual machine to be named eth0, eth1, ... based on their MAC address?

In bug 1046302 it says to use a different naming scheme than "eth". The CentOS 7 FAQ (and articles by Red Hat as well) tell you to use the boot options net.ifnames=0 biosdevname=0 to get the old naming scheme back.

When I did this, udev was unable to rename my devices to what I wanted, because the target name already existed, as mentioned in the bug report:

Jun 24 10:26:52 example systemd-udevd[388]: error changing net interface name eth0 to eth1: File exists

The fix was simple: use the new system together with udev rules.

  1. add net.ifnames=1 biosdevname=1 to GRUB_CMDLINE_LINUX in /etc/default/grub
  2. rebuild the GRUB configuration with grub2-mkconfig -o /boot/grub2/grub.cfg
  3. create /etc/udev/rules.d/70-persistent-net.rules and add some lines like
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:11:22:33:44:55", ATTR{type}=="1", KERNEL=="eth*", NAME="eth0"
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="22:33:44:55:66:77", ATTR{type}=="1", KERNEL=="eth*", NAME="eth1"
  1. reboot

In your boot logs you will see:

...
Jun 24 11:40:57 example systemd-udevd[271]: renamed network interface eth0 to ens256
...
Jun 24 11:40:59 example systemd-udevd[580]: renamed network interface ens256 to eth1

This has only been tested with VMware virtual machines, I don't know (yet) how it works on bare metal machines.

Tagged , , , , , , ,

EmulationStation: "Error initializing SDL!" on Raspberry Pi 2

I installed Arch Linux ARM on my rPi2, overwriting Raspbian. AUR has all the packages I need to build a game station myself. I built EmulationStation (git) with makepkg, but it wouldn't start:

[tristan@rpi2 ~]$ emulationstation 
lvl0:   Error initializing SDL!
        No available video device
lvl0:   Renderer failed to initialize!
lvl0:   Window failed to initialize!

Erwan35 found out the solution is to recompile SDL2 with OpenGL ES and Raspberry Pi 2 support. The [PKGBUILD](https://aur.archlinux.org/packages/sd/sdl2-opengles/PKGBUILD) for SDL2 with OpenGL ES support doesn't explicitly set the --host flag, and then the Raspberry Pi 2 is not detected properly.

Wrong ./configure output:

...
checking build system type... armv7l-unknown-linux-gnueabihf
checking host system type... armv7l-unknown-linux-gnueabihf
...

Make sure you run ./configure for SDL2 with the following options (update the PKGBUILD accordingly):

/configure --prefix=/usr \
    --enable-sdl-dlopen \
    --disable-arts --disable-esd --disable-nas \
    --enable-alsa \
    --disable-pulseaudio \
    --disable-video-wayland \
    --without-x --disable-video-x11 --disable-x11-shared \
    --disable-video-x11-xcursor --disable-video-x11-xinerama \
    --disable-video-x11-xinput --disable-video-x11-xrandr \
    --disable-video-x11-scrnsaver --disable-video-x11-xshape \
    --disable-video-x11-vm --disable-video-opengl \
    --disable-video-directfb --disable-rpath \
    --enable-video-opengles --host=arm-raspberry-linux-gnueabihf

Correct ./configure output:

...
checking build system type... armv7l-unknown-linux-gnueabihf
checking host system type... arm-raspberry-linux-gnueabihf
...

Install the newly built SDL2 and EmulationStation will launch fine:

[tristan@rpi2 sdl2-opengles]$ emulationstation
lvl0:   es_systems.cfg file does not exist!
lvl0:   Example config written!  Go read it at "/home/tristan/.emulationstation/es_systems.cfg"!
Tagged , ,

Raspbian root (/) on USB-stick for my Raspbian Pi 2

I recently bought a Raspberry Pi 2. At the same time I ordered a USB 3.0 flash disk, because I was afraid the SD-card would be too slow. I'm now going to put the root filesystem of Raspbian on the faster USB flash disk, while keeping the bootloader and /boot on the SD-card, because the rPi can't boot from USB.

Get the Raspbian image

My original idea was to use the Raspbian installer so I could install Raspbian like I'd install Debian. The installer on www.raspbian.org wouldn't boot on my rPi2. According to the timestamp, it was 2.5 years old. Maybe it doesn't support the rPi2 yet.

So instead, I downloaded the Raspbian image from the Raspberry Pi site, at the time the .zip file was 975 MB.

Inside the .zip file is the entire Raspberry OS, 3.1 GB in total. When you extract it, you end up with an .img file.

$  unzip 2015-02-16-raspbian-wheezy.zip 
Archive:  2015-02-16-raspbian-wheezy.zip
  inflating: 2015-02-16-raspbian-wheezy.img  

The USB-stick

Prepare the USB-stick

Insert the USB-stick in your computer and find out which device name it gets, using dmesg and lsblk.

Create a partition on your USB-stick for the root filesystem. My stick was 16 GB, so I used 15 GB for / and 1 GB for swap:

Disk /dev/sdc: 14,7 GiB, 15787360256 bytes, 30834688 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x000d032d

Device     Boot    Start      End  Sectors  Size Id Type
/dev/sdc1           2048 28733439 28731392 13,7G 83 Linux
/dev/sdc2       28735488 30832639  2097152    1G 82 Linux swap / Solaris

Copy / from .img to USB

The .img file contains two partitions, one is /boot and the second is /. The contents of the first go on the SD-card, and the other goes on the USB-stick.

The tricky part is dumping from the exact offset so you get all the files (blocks actually) you need, and nothing more. To figure out what the offset of the second partition is, use fdisk -l on the .img file:

$ fdisk -l 2015-02-16-raspbian-wheezy.img 

Disk 2015-02-16-raspbian-wheezy.img: 3,1 GiB, 3276800000 bytes, 6400000 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x0009bf4f

Device                          Boot  Start     End Sectors Size Id Type
2015-02-16-raspbian-wheezy.img1        8192  122879  114688  56M  c W95 FAT32 (LBA)
2015-02-16-raspbian-wheezy.img2      122880 6399999 6277120   3G 83 Linux

As you can see, the second partition starts at sector 122880. But dd, the tool we'll use to dump the data, uses bytes for its offset option. Luckily fdisk lists the sector size in its output:

Sector size (logical/physical): 512 bytes / 512 bytes

Simply multiply the starting sector of the second partition by 512 and you know the offset: 62914560 bytes!

To verify your calculation, use losetup to create a loopback device from the second partition inside the .img file:

 $ losetup --offset=62914560 /dev/loop0 /tmp/2015-02-16-raspbian-wheezy.img   

Now you can use dd to write the contents from the loopback device onto your USB-stick:

$ dd if=/dev/loop0 of=/dev/sdc1 bs=4M

Resize the USB disk

Mount the USB partition that holds the root filesystem for Raspbian we just dumped:

 $ mount /dev/sdc1 /mnt
 $ df -m /mnt
Filesystem     1M-blocks  Used Available Use% Mounted on
/dev/sdc1           2953  2421       364  87% /mnt

As you can see, the filesystem is only 3 GB in size, but the partition I created is 15 GB! That's because we used dd, to dump on a block level rather than a file level. To correct this, we just have to use resize2fs on the partition:

# umount /mnt
# fsck -fv /dev/sdc1 
fsck from util-linux 2.25.2
e2fsck 1.42.12 (29-Aug-2014)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
...
# resize2fs /dev/sdc1 
resize2fs 1.42.12 (29-Aug-2014)
Resizing the filesystem on /dev/sdc1 to 3591424 (4k) blocks.
The filesystem on /dev/sdc1 is now 3591424 (4k) blocks long.

# fsck -fv /dev/sdc1 
fsck from util-linux 2.25.2
e2fsck 1.42.12 (29-Aug-2014)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
...
root@thinker:~# mount /dev/sdc1 /mnt
root@thinker:~# df -m /mnt
Filesystem     1M-blocks  Used Available Use% Mounted on
/dev/sdc1          13745  2423     10605  19% /mnt

There you go: the entire 15 GB can be used by Raspbian now!

Update /etc/fstab on the USB-stick

While the USB-stick is still mounted, open /mnt/etc/fstab and update the line for the / file system from

/dev/mmcblk0p2  /               ext4    defaults,noatime  0       1

to

/dev/sda1       /               ext4    defaults,noatime  0       1

The SD-card

Prepare the SD-card

Create a single FAT32 partition on your SD-card. It doesn't have to be big, as we saw in the fdisk -l 2015-02-16-raspbian-wheezy.img output, the /boot partition is only 56 MB. Just make sure the partition is FAT32 (id C in fdisk) and that the boot flag is enabled.

Copy /boot from .img to SD-card

Mount the /boot partition from the .img file using the offset trick we saw before:

# mount -o loop,offset=$(expr 8192 \* 512) /tmp/2015-02-16-raspbian-wheezy.img /mnt/raspBoot

Mount your SD-card and copy the files required for /boot to the root of your SD-card. This time we use cp instead of dd because it's just a small amount of files:

# mount  mount /dev/mmcblk0p1 /mnt/sdCard
# cp -av /mnt/raspBoot/* /mnt/sdCard

Update boot parameters

Before unmounting the SD-card, open the file cmdline.txt and update the root= part so it looks like

dwc_otg.lpm_enable=0 console=ttyAMA0,115200 kgdboc=ttyAMA0,115200 console=tty1 root=/dev/sda1 rootfstype=ext4 rootwait text

Finishing up

Now umount both the USB-stick and the SD-card and plug them in your rPi2. Power on the rPi2 and it will boot from SD-card but the root file system will be on the USB-stick!

tristan@rpi2 ~ $ df -m
Filesystem     1M-blocks  Used Available Use% Mounted on
rootfs             13745  2424     10604  19% /
/dev/root          13745  2424     10604  19% /
devtmpfs             460     0       460   0% /dev
tmpfs                 93     1        93   1% /run
tmpfs                  5     0         5   0% /run/lock
tmpfs                391     0       391   0% /run/shm
/dev/mmcblk0p1       253    15       238   6% /boot
Tagged , ,

nginx User-Agent spoofing to circumvent LinkedIn's blacklisting

I have a few virtual hosts that redirect to my LinkedIn profile thanks to a rewrite rule in nginx:

server {
  server_name  example.com;
  listen       80;

  location / {
    rewrite    ^  http://be.linkedin.com/in/tristanterpelle/;
  }
}

This worked beautifully when I tested it, but today I noticed it was broken. LinkedIn returned a 999 error. After looking around a bit, it turns out that LinkedIn actively blocks HTTP requests from clients with certain User-Agent strings.

 $ curl -I --url https://be.linkedin.com/in/tristanterpelle 
HTTP/1.1 999 Request denied
$  curl -A "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.71 Safari/537.36" -I --url https://be.linkedin.com/in/tristanterpelle
HTTP/1.1 200 OK

OK, so curl's User-Agent is blacklisted, and apparently so is nginx' (wget also fails). Luckily, nginx can spoof its User-Agent thanks to the HttpHeadersMoreModule.

server {
  server_name  example.com;
  listen       80;

  location / {
    rewrite    ^  http://be.linkedin.com/in/tristanterpelle/;
    more_set_input_headers 'User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.71 Safari/537.36';
  }
}

On Debian, you need to install the nginx-extras package, or you will get a [emerg] unknown directive "more_set_input_headers" error.

Tagged , ,

VMware Workstation 9.0.1 on Debian with kernel 3.10

New major Linux kernel version, new major breakage of VMware Workstation. Here's what and how to patch to be able to compile VMware's modules so you can run your virtual machines under Linux kernel 3.10.

vmnet

The compilation of the vmnet.ko module fails, a patch found on the VMware Community forum fixes this.

It has to be applied to the contents of /usr/lib/vmware/modules/source/vmnet.tar. You can find the instructions on how to do this in my older post.

vmblock

After the first patch, we're still not out of the woods: the vmblock.ko won't compile either. In the same thread a link is provided to a blog post with the appropriate patch. This time, the patch has to be applied to the files in /usr/lib/vmware/modules/source/vmblock.tar.

Compile

Now you can compile the modules with vmware-modconfig --console --install-all, provided you installed the appropriate kernel header files for your kernel. In my case, the packages I needed were:

ii  linux-headers-3.10-2-amd64            3.10.7-1                         amd64        Header files for Linux 3.10-2-amd64
ii  linux-headers-3.10-2-common           3.10.7-1                         amd64        Common header files for Linux 3.10-2
ii  linux-image-3.10-2-amd64              3.10.7-1                         amd64        Linux 3.10 for 64-bit PCs

Sources

Tagged , , ,

Prey anti-theft on Debian

Anti-theft solution

Prey is an open source anti-theft solution. I decided to install it on my laptop so I could track it in case I lost it or it was stolen.

Prey is scheduled to "call home" to check the status of the device it's protecting. If it finds out the device went missing, it build a report with useful information to track down the device and tries to send it to the user. It can do the check and send the report via the Prey Control Panel (simple to setup) or you can run it in standalone mode which offers more options. I used the simple setup to find out if and how it works.

Installation

Version 0.5.3 is currently in the Debian repositories, which is easy to install (apt-get install prey) but after a few hours of trying, I couldn't get it to do anything useful: Prey noticed when my laptop was missing and created and sent the reports but they never arrived.

A post on Prey's anwers list recommended to upgrade to version 0.5.9 to get things working. I decided to install the latest version, which was 0.6.0. There's .deb for Ubuntu that I tried first on Debian Jessie/Sid, but long story short: it didn't work. Instead, I downloaded the universal .zip for Linux and made a .deb from it, the quick and dirty way.

Because it was built the quick and dirty way, Prey was complaining about a missing cronjob and configuration file.

The cronjob was in /etc/cron.d/prey when Prey expected it to be in root's list of crons. I copied the lines from /etc/cron.d/prey to crontab -e -u root and then rm'ed /etc/cron.d/prey. I changed the frequency to every 5 minutes.

Prey looked for the configuration file in /usr/lib/prey/config, which I solved by symlinking /etc/prey/config there: ln -sv /etc/prey/config /usr/lib/prey/config.

Like I said: a very quick and dirty packaging.

Configuration

To use Prey, you need to register an account. Once you've done that, log in and you will find your API key in the Account page. This key goes in your configuration file on the api_key= line. When you've set that, you can run Prey in test mode where it will check its status and generate a device key. It will immediately add the device key to the config file as well. Some moments later, your device will appear in Prey's Control Panel.

In Prey's Control Panel you can set the information you want gathered in the Devices tab. I deselected all the actions to perform on the right to keep Prey inconspicuous to a possible thief.

Testing

Set your device's status to Missing and wait for the cronjob to kick in. You should be getting a report with some very useful information a few minutes later. I checked and even when I didn't have X running, I got a nice report with a webcam picture.

Don't forget to set the status back to normal when you've verified that everything works: the free Prey account only has space for 10 reports.

Sources

Tagged , , ,