2008-06-03 16:43:08 Build process changes, 2006R??? to 2008R1-RC8
Graham Davies (UNITED STATES)
Message: 56641
We are in the process of migrating from 2006R-something-or-other to 2008R1-RC8 in order to pick up the fixed SD-Card on SPI stuff. It falls on my shoulders to figure out how to get the new source code safely tucked away in CVS so that we can tag and recover released versions in what I think of as the usual way.
When I did this for the old uClinux release, I had a lot of trouble finding out where the configuration files should be put so as not to have to fiddle with 'make menuconfig' on every fresh build. I eventually figured it out and reported back here:
This does not seem to work for the new uClinux release! Argh! I understand that uClibc has gone away, so that's one fewer configuration file to worry about. I can also see 'config.device' being copied from my 'vendor/board' directory to its place as '.config' in 'uClinux.dist'. But, one or both of the other two files (config.vendor-2.6.x and config.linux-2.6.x) are not being copied and in the output of 'make' I see what looks like a lot of configuring going on. I am trying to build, as before, with the following command executed in 'uClinux-dist' of a fresh export from CVS:
make <vendor>/<board>_default
Can anyone tell me what has changed in this area? It used to work so well.
Graham.
QuoteReplyEditDelete
2008-06-03 16:53:40 Re: Build process changes, 2006R??? to 2008R1-RC8
Mike Frysinger (UNITED STATES)
Message: 56642
the location of config files is documented in the wiki:
http://docs.blackfin.uclinux.org/doku.php?id=managing_kernel_upgrades
we dont store the kernel configs by default anymore in uclinux-dist ... but that is for our board configs. you can still store your kernel configs in uclinux-dist just fine.
QuoteReplyEditDelete
2008-06-03 17:15:31 Re: Build process changes, 2006R??? to 2008R1-RC8
Graham Davies (UNITED STATES)
Message: 56645
Mike,
OK, so that wiki page says I'm still looking at the same configuration files (uClibc excepted) and that they belong in the same place. There is nothing there, as far as I can see, about building without resorting to 'make menuconfig'. Has the ability to build everything directly from fresh source using a make target of the form <vendor>/<board>_default been removed from the 2008R1-RC8 distribution, please?
Graham.
QuoteReplyEditDelete
2008-06-03 17:25:46 Re: Build process changes, 2006R??? to 2008R1-RC8
Mike Frysinger (UNITED STATES)
Message: 56647
i rarely use menuconfig. i pretty much always use `make <vendor>/<board>_default`.
QuoteReplyEditDelete
2008-06-03 18:25:58 Re: Build process changes, 2006R??? to 2008R1-RC8
Graham Davies (UNITED STATES)
Message: 56648
So, I have the right configuration files in the right places and I'm using the right build command.
Can you tell me why, then, when I first build a fresh set of source exported from CVS the output of make looks as if configuration is taking place right there and then. After building config on the host, my make output looks like this:
scripts/kconfig/conf -o arch/blackfin/Kconfig
*
* Linux Kernel Configuration
*
*
* Code maturity level options
*
Prompt for development and/or incomplete code/drivers (EXPERIMENTAL) [Y/n/?] y
*
* General setup
*
Local version - append to kernel release (LOCALVERSION) [TEST] TEST
Automatically append version information to the version string (LOCALVERSION_AUTO) [Y/n/?] y
System V IPC (SYSVIPC) [Y/n/?] y
IPC Namespaces (IPC_NS) [N/y/?] n
POSIX Message Queues (POSIX_MQUEUE) [N/y/?] n
BSD Process Accounting (BSD_PROCESS_ACCT) [N/y/?] n
Export task/process statistics through netlink (EXPERIMENTAL) (TASKSTATS) [N/y/?] n
UTS Namespaces (UTS_NS) [N/y/?] n
Auditing support (AUDIT) [N/y/?] n
Kernel .config support (IKCONFIG) [Y/n/m/?] y
Enable access to .config through /proc/config.gz (IKCONFIG_PROC) [Y/n/?] y
Kernel log buffer size (16 => 64KB, 17 => 128KB) (LOG_BUF_SHIFT) [14] 14
Create deprecated sysfs files (SYSFS_DEPRECATED) [Y/n/?] y
... and so on.
This could be right for all I know. I'm asking for help because this is not what used to happen and I wan't to know if something has changed so that I should now expect this to happen. I want to be sure that the build is working properly. We are supposed to release this product on Friday and only today have I got a build from CVS to complete without error.
Graham.
QuoteReplyEditDelete
2008-06-03 19:05:06 Re: Build process changes, 2006R??? to 2008R1-RC8
Mike Frysinger (UNITED STATES)
Message: 56651
if your old config file doesnt match the new one (iow, you have missing options), then the kconfig process will launch to fill in the missing pieces. you need to update the defaults and then it should work fine.
QuoteReplyEditDelete
2008-06-04 09:21:34 Re: Build process changes, 2006R??? to 2008R1-RC8
Graham Davies (UNITED STATES)
Message: 56679
Mike, is there any way that you could give me some help on this issue? Normally, I would just spend however long it took to figure it out, but now we are up against a deadline and have almost no time left to test the final firmware with the new version of uClinux. As far as I'm concerned, I don't have an 'old config' or a 'new one'. I don't want 'defaults', I want the stuff I've explicitly chosen. I have shown you that I understand which files store my configuration and where they belong in the vendor/board directory. I have shown you that I know how to start a build from a fresh CVS checkout of the source. Can you please tell me how to have that build happen without what appears to be a configuration taking place at the start of the build?
Graham.
QuoteReplyEditDelete
2008-06-04 15:28:33 Re: Build process changes, 2006R??? to 2008R1-RC8
Graham Davies (UNITED STATES)
Message: 56690
Is anyone else out there building their 2008R1-RC8 distribtion with 'make <vendor>/<board>_default'?
When you do this for the first time, i.e. with a fresh copy of the sources, does the output of the build suggest that configuration is taking place? That is, do you get lots of lines ending with [Y/n/?] y and [Y/n/?] n, etc.?
If not, can you give me any information about how you worked this? My guess is that it has something to do with preparing the configuration files in the <vendor>/<board> directory.
Graham.
QuoteReplyEditDelete
2008-06-04 16:24:31 Re: Build process changes, 2006R??? to 2008R1-RC8
Mike Frysinger (UNITED STATES)
Message: 56691
if the kconfig system is asking you questions, you're using an old config. that's simply how it works. you need to take your old default configs, run them through the configuration process once, and then update your defaults using those files.
QuoteReplyEditDelete
2008-06-04 17:13:18 Re: Build process changes, 2006R??? to 2008R1-RC8
Graham Davies (UNITED STATES)
Message: 56700
Mike, this is the same answer as before. I don't understand what you mean by "old config". I have one single set of config files tucked away in my <vendor>/<board> directory. These were produced by the person who configured uClinux 2008R1-RC8 and whose changes I now have to get into CVS along with the distribution he started from. He did not build using 'make <vendor>/<board>_default', instead he ran 'make menuconfig', wandered around the menus a bit and then used just 'make'. This is not what I want to preserve as the way to re-create released firmware versions from tagged source in CVS. I am trying to get 'make <vendor>/<board>_default' to work.
I see that in my uClinux-dist directory I have a file called '.oldconfig'. Is this what you mean?
The text 'oldconfig' appears three times in the output of make. Should I be looking for a clue here?
When you say "[my] defaults", are these the config files in my <vendor>/<board> directory? So, does "take [my] old default configs ..." mean you want me to read my configuration files in with menuconfig, write them out again and then copy the resulting files back into my <vendor>/<board> directory? This doesn't make sense because that's how they were created in the first place, but I will give it a try.
Graham.
QuoteReplyEditDelete
2008-06-04 18:01:19 Re: Build process changes, 2006R??? to 2008R1-RC8
Mike Frysinger (UNITED STATES)
Message: 56707
the reason it works by doing `make menuconfig` is that all of the old->new transition is handled automatically by the gui based on the default values encoded into the kconfig files. when you use the _default method, no gui is there to handle the transition for you, thus it prompts you with all the questions.
the default settings live in vendors/<vendor>/<board>/. everything else (like .config) are your active configuration files.
there is no need for you to copy files by hand ... the menuconfig system already has an option for doing just that:
[ ] Update Default Vendor Settings
in fact, this exact process should already be described in the document i pointed out originally:
if there is something in that that needs clarification, let me know and i'll update it
QuoteReplyEditDelete
2008-06-05 10:33:57 Re: Build process changes, 2006R??? to 2008R1-RC8
Graham Davies (UNITED STATES)
Message: 56735
I am not involved in a "old->new transition" (if I understand this correctly). I have configuration files created by someone else, most probably using 'make menuconfig' and also most probably from scratch, as there seem to be many differences between the old and new systems.
My problem is that when I park those configuration files in what I think we agree is the right place and use what I think we agree is the right command to start a build from a fresh export of source from CVS, the output of the build process contains evidence of adjustments to the configuration.
As this thread keeps returning to advice I understand but that doesn't help, I conclude that we've turned wrong corner and will have to figure this out for ourselves. Although it makes no sense to me, I will try reading the configuration files one more time into 'menuconfig', spitting them out again and then replacing the files in <vendor>/<board> with the new files.
Graham.
QuoteReplyEditDelete
2008-06-05 11:00:34 Re: Build process changes, 2006R??? to 2008R1-RC8
Bernd Schmidt (GERMANY)
Message: 56743
Maybe if you posted the exact steps you perform, the exact messages you see, and the exact config files you're using, we could help you better. In effect, you're keeping us guessing as to what's going on at your end.
QuoteReplyEditDelete
2008-06-06 16:15:52 Re: Build process changes, 2006R??? to 2008R1-RC8
Graham Davies (UNITED STATES)
Message: 56831
Well, I *have* posted steps and messages, but the thread is in such a snarl I think I need to start again. Also, we missed the product release, so I have a brief reprieve in which to figure this out.
(1) - I think we are agreed that `make <vendor>/<board>_default` is the correct way to build a fresh checkout of source files using a previously established configuration that has been tucked away in the `<vendor>/<board>` directory. This has worked for me for a year with uClinux 2006-something-or-other. Mike also says that he does this, presumably with 2008R1-RC8, although nobody has actually said yes, this is the right thing to do in this situation.
(2) - The next problem is how to tuck away the configuration files. I'm assuming that the uClibc configuration file from 2006-something-or-other has gone away and that the other three files serve the same purpose in 2008R1-RC8, so they are:
uClinux-dist/.config <----------------> uClinux-dist/vendors/<vendor>/<board>/config.device
uClinux-dist/config/.config <-------> uClinux-dist/vendors/<vendor>/<board>/config.vendor-2.6.x
uClinux-dist/linux-2.6.x/.config <--> uClinux-dist/vendors/<vendor>/<board>/config.linux-2.6.x
Again, nobody will say that this is right or wrong. I'm still hoping that it is right.
(3) - Below are the outputs of `make <vendor>/<board>_default`, up to where we dive into the linux-2.6.x diectory, for the OLD (2006-something-or-other) and then the NEW (2008R1-RC8) distributions of uClinux:
make distclean > /dev/null 2>&1
cp vendors/MyVendor/MyBoard/config.device .config
cp vendors/MyVendor/MyBoard/config.uClibc uClibc/.config
cp vendors/MyVendor/MyBoard/config.vendor-2.6.x config/.config
cp vendors/MyVendor/MyBoard/config.linux-2.6.x linux-2.6.x/.config
ln -sf vendors/MyVendor/MyBoard/config.arch .
yes "" | make oldconfig > /dev/null 2>&1
make
make[1]: Entering directory `/home/graham/OLD/uClinux-dist'
make ARCH=blackfin CROSS_COMPILE=bfin-uclinux- -C linux-2.6.x || exit 1
make[2]: Entering directory `/home/graham/OLD/uClinux-dist/linux-2.6.x'
make clean > /dev/null 2>&1
cp vendors/MyVendor/MyBoard/config.device .config
chmod u+x config/setconfig
yes "" | config/setconfig defaults
make[1]: Entering directory `/home/graham/NEW/uClinux-dist'
KCONFIG_NOTIMESTAMP=1 make ARCH=blackfin CROSS_COMPILE=bfin-uclinux- -C linux-2.6.x oldconfig
make[2]: Entering directory `/home/graham/NEW/uClinux-dist/linux-2.6.x'
I understand why config.uClibc is now not being copied up from <vendor>/<board>. I don't understand why config.vendor-2.6.x and config.linux-2.6.x are not being copied up from <vendor>/<board>. I'm uncomfortable that later on in the output from make (posted at 2008-06-03 18:25:58) there appears to be some configuration taking place. This gets me back to my first post in the thread. Can someone explain to me what has changed in the build process and how I can get back to the situation I was in before?
Graham.
QuoteReplyEditDelete
2008-06-06 20:43:52 Re: Build process changes, 2006R??? to 2008R1-RC8
Bernd Schmidt (GERMANY)
Message: 56835
You can probably find the change you're worried about in svn with
svn diff -r 4805:4806
You wrote:
I don't understand why config.vendor-2.6.x and config.linux-2.6.x are not being copied up from <vendor>/<board>
This now appears to be the task of the setconfig script. Change the first line in it to "#! /bin/sh -x" to see what it's doing. It should detect the kernel version and then install a configuration file. I'm testing with BF548-EZKIT, and the directory doesn't contain a config-linux.2.6.x, so it uses one of the kernel defconfigs instead:
cp linux-2.6.x/arch/blackfin/configs/BF548-EZKIT_defconfig linux-2.6.x/.config
Previously, the Makefile contained this:
yes "" | make oldconfig > /dev/null 2>&1
The setconfig script now just runs "make oldconfig_linux" without redirecting the output to /dev/null, which is why you see output. So, that's what seems to have changed. Presumably, "make oldconfig" would stop and prompt you if the configs you're using had a problem or were out-of-date.
QuoteReplyEditDelete
2008-06-09 11:36:34 Re: Build process changes, 2006R??? to 2008R1-RC8
Graham Davies (UNITED STATES)
Message: 56873
Bernd, Thank you very much for stepping in to help me with my problem. From your reply, I see that there have been substantial changes to the build process that do explain the difference in the output I see now from what I saw before. I am not able to figure out how to get the svn diff you mention, but I have looked at our local CVS diffs of the files you pointed me to between our 2006 distribtion and 2008R1-RC1 and tried your suggestion to make visible what the setconfig script is doing.
Although I can't say I fully understand what is going on, my comfort level has increased to the point that I'm prepared to tell management here that the build is working and we can release firmware built from our CVS repository in the same way as before.
In case this helps anyone else, here is a summary:
In the top-level makefile, the %_default target used to unconditionally copy all configuration files up from the <vendor>/<board> directory and this was clearly visible in the build output. Now, that does not happen. Instead, the script config/setconfig is responsible for this, on one of the two occasions on which it is called from the Makefile, using considerably more complex decision logic. The output of the script does not appear in the build output, so direct evidence of the configuration file copy is missing.
In the top-level makefile, the %_default target used to perform a `make oldconfig` piping in return characters to answer questions (I think) and piping both STDOUT and STDERR to /dev/null. Thus, the question-and-answer session that I presume resulted from this did not appear in the build output. Now, config/setconfig also takes over this job, but performs a `make oldconfig_linux` instead (which I presume amounts to the same thing). The output is *not* piped to /dev/null, so the question-and-answer session appears in the build output.
These two changes drastically alter the build output, hiding the copying up of the <vendor>/<board> configuration files and exposing a question-and-answer session that makes it appear that configuration information is missing and is being generated on-the-fly. Nevertheless, I now see that what is actually happening is very probably the same as before.
Once again, many thanks to Bernd.
Graham.